alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
2,457 Commits 3 Branches 0 Tags 51 MiB
8fefac635c
Commit Graph

741 Commits

Author SHA1 Message Date
Jiri Pechanec
0e98bb428b DBZ-978 Make offset readable in last event 2018-12-19 11:03:01 +01:00
luobo
fcfc019b6b fix the wrong words 2018-12-14 13:05:14 +01:00
Gunnar Morling
47f22feddc DBZ-978 Misc. clean-up 2018-12-13 21:17:23 +01:00
Jiri Pechanec
edec1c3090 DBZ-978 Added last event and captured tables metrics 2018-12-13 20:52:11 +01:00
Gunnar Morling
f89fc0bb14 DBZ-966 Awaiting snapshot delay only when actually doing a snapshot 2018-12-13 14:29:54 +01:00
Gunnar Morling
5542c0de29 DBZ-966 Making sure Postgres connector can be stopped while awaiting initial snapshot delay; 
* Also skipping snapshotting when requested to stop during initial delay
* Showing remaining delay time in logs
* Adding missing config
 2018-12-13 14:29:54 +01:00
Gunnar Morling
7a577161da DBZ-966 Pushing snapshot delay logic into coordinator; 
Using timer + metronome to ensure the delay phase will be cancelled upon connector shutdown
 2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
7f8c0e8ff1 DBZ-966 Enable to delay initial snapshot in HistorizedRelationalSnapshotChangeEventSource 2018-12-13 14:29:54 +01:00
Grzegorz Kołakowski
2a1934eeb1 DBZ-966 Make snapshot.delay.ms global property 2018-12-13 14:29:54 +01:00
Jiri Pechanec
2b2ff0754d DBZ-960 Run SnapshotReader in a single transaction 2018-12-11 14:15:53 +01:00
Jiri Pechanec
284ae98adf DBZ-1003 Initialize database history 2018-12-03 21:52:05 +01:00
Renato Mefi
8c1ebc2a8f DBZ-971 Unwrap can add data operation header 2018-11-30 12:06:18 +01:00
Renato Mefi
0af8dcfde0 DBZ-971 Ensure Envelope unwrap propagates headers 2018-11-30 12:06:18 +01:00
Jenkins user
db1d3a7fb8 [maven-release-plugin] prepare for next development iteration 2018-11-20 16:15:14 +00:00
Jenkins user
f83db82cea [maven-release-plugin] prepare release v0.9.0.Beta1 2018-11-20 16:15:14 +00:00
Gunnar Morling
6bd4024618 DBZ-776 Misc. improvements; 
* Renaming getTimeSinceLastEvent() to getMilliSecondsSinceLastEvent()
* Further unifying metrics implementation across connectors
* Emitting event in EventDispatcher also if event is filtered out
* Typo fixes
 2018-11-16 08:52:07 +01:00
Jiri Pechanec
4510e2bbe6 DBZ-776 Cleanup after review 2018-11-16 08:52:07 +01:00
Gunnar Morling
360714a5a1 DBZ-776 Formatting and minor clean-up 2018-11-16 08:52:07 +01:00
Jiri Pechanec
b44c9fd04f DBZ-776 A first set of metrics implemented in new framework 2018-11-16 08:52:07 +01:00
Jiri Pechanec
4cdc3e669b DBZ-776 Extract metrics to a common code 2018-11-16 08:52:07 +01:00
Gunnar Morling
9436778a1d DBZ-812 Misc. clean-up 2018-11-13 10:06:22 +01:00
Jiri Pechanec
33f6529614 DBZ-812 PK list is not-null for no PKs on table 2018-11-13 09:41:03 +01:00
Jiri Pechanec
5930c580f7 DBZ-812 Change extract column metadata 2018-11-13 09:41:03 +01:00
Jiri Pechanec
839d7c80eb DBZ-812 Prep statements with different parameters; Extract table schema code 2018-11-13 09:41:03 +01:00
Gunnar Morling
92704ce394 DBZ-963 Removing hard-coded references to "target" dir; 
Also injecting actual build dir via system variable.
Making DATA_DIR a simple string.
 2018-11-08 10:35:07 +01:00
Anton Martynov
3401cd23ea DBZ-963 Allowing to customize the directory where tests are storing their data 2018-11-08 10:34:10 +01:00
Gunnar Morling
9994172860 DBZ-953 Pulling decimal mode handling up to RelationalDatabaseConnectorConfig; 
Also using it in MySQL connector.
 2018-11-08 09:23:28 +01:00
Grzegorz Kołakowski
2d5295805c DBZ-941 Add connectionCreated template method in HistorizedRelationalSnapshotChangeEventSource 
The method allows to define steps which have to be taken just after the
database connection is created (e.g. setting snapshot isolation level).

By default no operation is executed.
 2018-10-30 13:35:57 +01:00
Jenkins user
d952f5dfb0 [maven-release-plugin] prepare for next development iteration 2018-10-04 11:59:14 +00:00
Jenkins user
ff9b70278b [maven-release-plugin] prepare release v0.9.0.Alpha2 2018-10-04 11:59:13 +00:00
jchipmunk
05ba99bb0a DBZ-918 Adding Debezium connector field to source info 
This will allow consumers to recognize the Debezium connector used for creating a given message, helping them to adjust their behavior for a variety of connectors.
 2018-10-04 09:00:13 +02:00
Gunnar Morling
39a47fabfa DBZ-899 Using custom statement to obtain SQL type info in order to avoid N+1 issues with PgDatabaseMetaData#getTypeInfo() 2018-09-19 09:51:38 +02:00
Gunnar Morling
3331d4c1a9 DBZ-899 Removing unused field; 
Also inlining method only used in one place.
 2018-09-19 09:51:38 +02:00
Jiri Pechanec
2253eaa6a6 DBZ-819 JdbcConnection is not thread safe 2018-09-18 09:05:36 +02:00
Jiri Pechanec
05ce2e3063 DBZ-819 Cache prepared statements in connection 2018-09-18 09:05:36 +02:00
jchipmunk
b0feaf496a DBZ-633 Adding "field.blacklist" property for MongoDB connector 
The "field.blacklist" configuration property is an optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are of the form "databaseName.collectionName.fieldName.nestedFieldName", where "databaseName" and "collectionName" may contain the wildcard (*) which matches any characters.

Although the "field.blacklist" configuration property allows you to remove fields from the event values, the "_id" field is always included in the event’s key.
 2018-09-14 12:54:11 +02:00
Gunnar Morling
5878e5105f
DBZ-878 Using LRU eviction strategy 
[As recommended](4961277637 (commitcomment-30412629)) by Ben Manes, the LIRS implementation seems to provide no advantage and isn't well tested.
 2018-09-06 11:21:31 +02:00
Gunnar Morling
4961277637 DBZ-878 Making topic name cache size bound; 
* Imported BoundedConcurrentHashMap from Hibernate ORM as a thread-safe concurrent cache.
 2018-09-05 12:56:53 +02:00
Gunnar Morling
e1bbbd7992 DBZ-865 DBZ-878 Replacing invalid topic name characters with "_"; 2018-09-05 12:56:53 +02:00
Gunnar Morling
eba134817f DBZ-894 Making TableIdToStringMapper dedicated functional interface with specifically named method 2018-09-04 12:43:50 +02:00
Jiri Pechanec
1373ac8475 DBZ-894 Introduce explicit mapping functional type 2018-09-04 12:39:10 +02:00
Jiri Pechanec
3e0648a42c DBZ-894 Customize TableId comparison per-connector in filters 2018-09-04 09:18:15 +02:00
Gunnar Morling
985999b4af DBZ-872 Making MySqlAntlrDdlParser#tableFilter final 2018-08-24 08:12:54 +02:00
Jiri Pechanec
63ee65bb7d DBZ-872 Honor filters in ALTER TABLE parsing 2018-08-24 08:12:54 +02:00
Jiri Pechanec
82a93cdac5 DBZ-859 Heartbeat is sent after pipeline snapshot 2018-08-23 11:44:53 +02:00
Jiri Pechanec
985e03e7b5 DBZ-859 Force heartbeat after snapshot completion 2018-08-23 11:44:53 +02:00
Maciej Bryński
947abf9815 DBZ-857 Ability to rewrites deleted records 2018-08-14 10:37:39 +02:00
artiship
57fa31e1bc DBZ-854 Correct param name for excludeColumns(String fullyQualifiedTableNames) 2018-08-14 08:39:28 +02:00
LiuHanlin
41e16f4a4f DBZ-853 Fix kafka database history storage miscount attemp number even if there are more records to consume 2018-08-13 20:26:48 +02:00
Jenkins user
e00dad127f [maven-release-plugin] prepare for next development iteration 2018-07-26 08:00:12 +00:00
Jenkins user
16bfd5c700 [maven-release-plugin] prepare release v0.9.0.Alpha1 2018-07-26 08:00:12 +00:00
Gunnar Morling
3061c1dc0b DBZ-815 Passing offset to Heartbeat#heartbeat(); 
That's simpler to grasp than the approach of passing a supplier lambda to the constructor.
Also it allows to pass on the offset via local variables instead of instance fields in some cases.
 2018-07-25 14:49:52 +02:00
Gunnar Morling
0a659c0159 DBZ-815 Ensuring latest offset is propagated also if change events relate to non-whitelisted tables 2018-07-25 14:49:52 +02:00
Gunnar Morling
fa68c1a158 DBZ-816 Preparing code for database history set-up for re-use 2018-07-25 11:54:15 +02:00
Jiri Pechanec
e86ddc0ab2 DBZ-40 Add blocking version of MultiResultSetConsumer 2018-07-25 11:51:09 +02:00
Gunnar Morling
bd4f3d60fa DBZ-804 Simplifying construction of VariableScaleDecimal 2018-07-24 20:48:52 +02:00
Gunnar Morling
98c329aabe DBZ-40 Adding SnapshottingTask#toString() 2018-07-24 18:53:32 +02:00
Gunnar Morling
0362b333bf DBZ-804 Better schema comparison in tests; preparing byte conversion 2018-07-24 05:18:52 +02:00
Gunnar Morling
9814a5c82c DBZ-803 Finalizing snapshot in offsets also for schema-only snapshots 2018-07-23 06:44:04 +02:00
Gunnar Morling
2905cc498b DBZ-800 Don't expect that partition is a Map<String, String> 2018-07-20 13:37:54 +02:00
Gunnar Morling
fe53360f30 DBZ-800 Avoiding "mysql" in commonly used heartbeat key schema 2018-07-20 13:37:54 +02:00
Gunnar Morling
86a4731587 DBZ-800 Emitting heartbeat events in generic event dispatcher 2018-07-20 13:37:54 +02:00
Gunnar Morling
26106cdd8e DBZ-800 Removing unused method from Heartbeat 2018-07-20 13:37:54 +02:00
Jiri Pechanec
0e35439019 DBZ-40 Snapshotting and connector restart 2018-07-18 14:39:11 +02:00
Gunnar Morling
a5b5e7d9eb DBZ-627 Using unfified TopicSelector for MongoDB, too 2018-07-18 12:36:07 +02:00
Gunnar Morling
20db9299c5 DBZ-627 Unifying TopicSelector implementations 2018-07-18 12:36:07 +02:00
Gunnar Morling
610da70cc5 DBZ-801 Parameterizing ChangeEventQueue<DataChangeEvent> 2018-07-18 10:30:22 +02:00
Gunnar Morling
80085567ae DBZ-720 Proper check for previous unfinished snapshot 2018-07-18 10:07:50 +02:00
Gunnar Morling
33170d4c3d DBZ-40 Pulling up some common code to HistorizedRelationalDatabaseSchema 2018-07-18 10:07:50 +02:00
Gunnar Morling
422a303f87 DBZ-720 Signaling cancellation of snapshot through InterruptedException 2018-07-17 13:11:02 +02:00
Jiri Pechanec
f118e37da0 DBZ-40 Changes to support SQL Server connector 2018-07-13 22:23:00 +02:00
Gunnar Morling
c5e2d5f9e7 DBZ-720 Making scan of single table interruptable; 
Also better logging.
 2018-07-13 19:42:18 +02:00
Gunnar Morling
648d695c08 DBZ-720 Preparing HistorizedRelationalSnapshotChangeEventSource to support data snapshotting 2018-07-13 19:42:18 +02:00
Gunnar Morling
aa1272cc21 DBZ-720 Re-using single instance of streaming event receiver in EventDispatcher 2018-07-13 09:51:15 +02:00
Gunnar Morling
657650c581 DBZ-720 Passing change event emitter directly to dispatch methods; 
The indirection of going through a supplier wasn't really necessary
 2018-07-13 09:51:15 +02:00
Gunnar Morling
a565361e56 DBZ-720 Making SnapshotContext auto-closeable 2018-07-13 09:51:15 +02:00
Gunnar Morling
986df43649 DBZ-720 Extracting HistorizedRelationalSnapshotChangeEventSource from Oracle connector 2018-07-13 09:51:15 +02:00
Gunnar Morling
1523f230ca DBZ-793 Using TableFilter consistently; 
Only the MySQL connector's Filters class still uses Predicate for the time being, so to avoid to much merging trouble with DBZ-175.
 2018-07-11 13:16:44 +02:00
Jenkins user
f9b8d830a8 [maven-release-plugin] prepare for next development iteration 2018-07-11 07:36:30 +00:00
Jenkins user
290ded678f [maven-release-plugin] prepare release v0.8.0.Final 2018-07-11 07:36:29 +00:00
Jenkins user
033db6659d [maven-release-plugin] prepare for next development iteration 2018-07-04 07:07:44 +00:00
Jenkins user
696f35f2c0 [maven-release-plugin] prepare release v0.8.0.CR1 2018-07-04 07:07:44 +00:00
Gunnar Morling
99048365c9 DBZ-771 Unsetting default value when changing a column's type 2018-07-03 11:55:20 +02:00
Gunnar Morling
c7d5288d40 DBZ-771 Making sure causing DDL statement is logged also if default value can't be parsed 2018-07-03 11:55:20 +02:00
Gunnar Morling
dcd6b01134 DBZ-644 Simplifying RelationalDatabaseSchema constructor; 
Also retrieving logical name via CommonConnectorConfig#getLogicalName() in more places
 2018-07-03 07:03:59 +02:00
Gunnar Morling
b0980b994c DBZ-644 Adding test for Postgres 2018-07-03 07:03:59 +02:00
Gunnar Morling
068aa85bd6 DBZ-644 Misc. clean-up and tests; 
* Using parameter keys under "__debezium" namespace
* More expressive names
* Adding tests
 2018-07-03 07:03:59 +02:00
Gunnar Morling
8744488c8a DBZ-644 Unifying retrieval of ColumnMappers in RelationalDatabaseSchema 2018-07-03 07:03:59 +02:00
orr.ganani
ac515b8064 DBZ-644 change Original Data Type mapper and add a test 2018-07-03 07:03:59 +02:00
orr.ganani
747184c572 DBZ-644 add JDBC data type as additional metadata to DBZ events, if configuration is opted in 2018-07-03 07:03:59 +02:00
Gunnar Morling
c4c8cbc3ab DBZ-773 Moving management of Tables to RelationalDatabaseSchema 2018-06-29 09:09:10 +02:00
Gunnar Morling
bbfbdf6fab DBZ-773 Moving management of table schemas to RelationalDatabaseSchema 2018-06-29 09:09:10 +02:00
Gunnar Morling
a292b05a96 DBZ-751 Propagating DECIMAL column precision to Avro schemas 2018-06-27 14:48:59 +02:00
Gunnar Morling
b27421ddd0 DBZ-759 Testing via SchemaUtil; 
* Making RecordWriter private
* Removing quotes around string representation of byte arrays
 2018-06-26 12:03:53 +02:00
Andreas Bergmeier
82bac78ba8 DBZ-759 Add Test to ensure correct serialization of byte arrays 
Needed to make RecordWriter public to have access in tests.
 2018-06-26 11:50:54 +02:00
Andreas Bergmeier
71abca2afb DBZ-759 Fix unusable array information in Serialization 
According to ErrorProne, (implicitly) calling toString on the array does
not give useful information.
Also applying Base64 encoding is strange here.
 2018-06-26 11:50:54 +02:00
Gunnar Morling
8d0d35762e DBZ-759 Removing superfluous (and broken) ColumnEditorImpl#compareTo() implementation 2018-06-25 17:04:45 +02:00
Gunnar Morling
bf7a5018ca Setting POM version back to 0.8.0-SNAPSHOT 2018-06-22 12:21:03 +02:00
Gunnar Morling
357102158a DBZ-759 Fixing broken Document#setArray() method 2018-06-22 11:03:00 +02:00
Gunnar Morling
ce81f37990 DBZ-759 Adding unit test 2018-06-22 09:19:45 +02:00
Andreas Bergmeier
7417baea4d DBZ-759 Fix equality comparison for BinaryValue 
If value is binary or string, it should be better to compare the content
of the actual byte arrays.
According to ErrorProne you use reference equality when calling equals on
an array.

Part of https://issues.jboss.org/browse/DBZ-759
 2018-06-22 09:19:45 +02:00
Andreas Bergmeier
333d6cb57d DBZ-759 Fix hashCode of BinaryValue 
According to ErrorProne, the hashCode of a byte array returns useless
information.

Part of https://issues.jboss.org/browse/DBZ-759
 2018-06-22 09:19:41 +02:00
Andreas Bergmeier
d374ca0464 DBZ-759 Fix ColumnImpl.scale equality check 
According to ErrorProne, checking Optional via operator == leads to
reference equality check. Probably want rather to check for contained
values.
 2018-06-22 09:08:27 +02:00
Jenkins user
db42e4657a [maven-release-plugin] prepare for next development iteration 2018-06-21 14:07:45 +00:00
Jenkins user
c4b8ecaa99 [maven-release-plugin] prepare release v0.8.0.Beta1 2018-06-21 14:07:45 +00:00
Gunnar Morling
96c59a1568 DBZ-20 Moving debezium-connector-oracle from main to incubator repo 2018-06-20 13:05:37 +02:00
Gunnar Morling
b83f1493a2 DBZ-20 Avoiding Optional as method parameter 2018-06-20 13:05:37 +02:00
Gunnar Morling
dce1aa29f1 DBZ-20 Oracle precision supports negative numbers 2018-06-20 13:05:37 +02:00
Gunnar Morling
f4e3668ebf DBZ-20 Tune up timeouts/connector/oracle/OracleStreamingChangeEventSource.java 2018-06-20 13:05:37 +02:00
Gunnar Morling
cf6219ccda DBZ-20 Committing processed offsets in the DB, allowing log files to be released 2018-06-20 13:05:37 +02:00
Jiri Pechanec
661c34fc6b DBZ-20 Use fluent API in ColumnEditor 2018-06-20 13:05:37 +02:00
Jiri Pechanec
232a1d5573 DBZ-20 Support for day to second interval 2018-06-20 13:05:37 +02:00
Jiri Pechanec
89694ffbbf DBZ-20 Support for year to month interval 2018-06-20 13:05:37 +02:00
Jiri Pechanec
28b844bf99 DBZ-20 Added ANSI double precision, real and variable scale number datatypes 2018-06-20 13:05:37 +02:00
Jiri Pechanec
10ddb69dea DBZ-20 A subset of string, time and number datatypes 2018-06-20 13:05:37 +02:00
Gunnar Morling
603f02e70d DBZ-20 Excluding generated sources from CheckStyle 2018-06-20 13:05:37 +02:00
Gunnar Morling
e8614c7890 DBZ-20 Adding table whitelist/blacklist connector options; 
* Restructuring OracleSnapshotChangeEventSource into smaller methods
 2018-06-20 13:05:37 +02:00
Gunnar Morling
1e87fae023 DBZ-20 Recovering database history 2018-06-20 13:05:37 +02:00
Gunnar Morling
0733110696 DBZ-20 Recording schema history; 
This records the DDL for DDL events captured during streaming. For the
initial schema snapshot, a JSON-style representation of the captured
Table objects is used in a new field of HistoryRecord, as the DDL
returned by dbms_metadata.get_ddl() isn't fully parseable by our
grammar.
 2018-06-20 13:05:37 +02:00
Gunnar Morling
1a7786d254 DBZ-20 Further simplifying threading model and stopping logic 2018-06-20 13:05:37 +02:00
Gunnar Morling
d8a9865e51 DBZ-20 Some more docs 2018-06-20 13:05:37 +02:00
Gunnar Morling
225c0f05a6 DBZ-20 Extracting RelationalChangeRecordEmitter base class 2018-06-20 13:05:37 +02:00
Gunnar Morling
979c60d956 DBZ-20 Handling update events 2018-06-20 13:05:37 +02:00
Gunnar Morling
24e22f4985 DBZ-20 Simpler and more robust interruption handling in ChangeEventSourceCoordinator 2018-06-20 13:05:37 +02:00
Gunnar Morling
d7e196a18e DBZ-20 Initial import of Oracle connector based on XStream 2018-06-20 13:05:37 +02:00
Gunnar Morling
9eb4b90ec9 DBZ-191 Adding record validations 2018-06-15 20:23:17 +02:00
Gunnar Morling
84dd36df46 DBZ-252 Misc typo fixes and clean-up 2018-06-15 16:43:30 +02:00
Gunnar Morling
54ca30624d DBZ-252 Misc. improvements; 
* Dedicated getter for DDL mode
* Using Objects#equal()
* Typo fixes
 2018-06-15 16:43:29 +02:00
Jiri Pechanec
9580c8c290 DBZ-252 Rebase to master 2018-06-15 11:42:24 +02:00
rkuchar
7d7f740721 DBZ-252 refactor parser listener 2018-06-15 11:42:24 +02:00
rkuchar
5c9e718436 DBZ-252 uncomment getting ddl parser instance from configuration + improve/refactor dataType resolver for ANTLR 2018-06-15 11:42:24 +02:00
rkuchar
99eac9a8e2 DBZ-252 some test fixes + separate ddl test for antlr parser 2018-06-15 11:42:24 +02:00
rkuchar
22151250fa DBZ-252 refactor proxy listener delegation 2018-06-15 11:42:24 +02:00
rkuchar
07212043b5 DBZ-252 test fixes + grammar fixes + some bug repairs in antlr parser 2018-06-15 11:42:23 +02:00
rkuchar
1124f81d94 DBZ-252 parse set and use statements 2018-06-15 11:42:23 +02:00
rkuchar
b4480c55b7 DBZ-252 parse database queries (create, alter, delete), parse name tokens without quotes, add new Mysql connector configuration field for choosing ddl parsing mode 2018-06-15 11:42:23 +02:00
rkuchar
5d911d511f DBZ-252 signal event for truncate table into schema changes 2018-06-15 11:42:23 +02:00
rkuchar
32d181480d DBZ-252 fix troubles with generalized system variables + add new data types that was missing in grammar 2018-06-15 11:42:23 +02:00
rkuchar
85e8146d1f DBZ-252 implementation of scope synonym for local and session scope in MySQL 2018-06-15 11:42:23 +02:00
rkuchar
5851d1f07d DBZ-252 fix initialization of system variables object + add default scope for variales in case that scope will no be set 2018-06-15 11:42:23 +02:00
rkuchar
9311c82bc5 DBZ-252 reformat code to not have anything after closing "}" 2018-06-15 11:42:23 +02:00
rkuchar
782ab75160 DBZ-252 move listener support from base to only legacy implementation, refactor system variables for possible use with other DBMSs + introduce data type resolver for antlr mysql parser 2018-06-15 11:42:23 +02:00
rkuchar
b92e8ef760 DBZ-252 exclude logging of parsed comment 2018-06-15 11:42:23 +02:00
rkuchar
30e158aa71 DBZ-252 create table and drop table listeners 2018-06-15 11:42:23 +02:00
rkuchar
785c346bcb DBZ-252 Exclude reusable logic from LegacyDdlParser 2018-06-15 11:42:23 +02:00
rkuchar
36e2f33bf6 DBZ-252 - extract creation of DdlParser interface 2018-06-15 11:42:23 +02:00
Jiri Pechanec
29f8891b4f DBZ-578 Process timestamps without timezones as in server timezone 2018-06-11 15:27:08 +02:00
Jiri Pechanec
de0e1ad4b5 DBZ-702 Mask SASL configuration 2018-06-08 08:57:50 +02:00
Jiri Pechanec
176dcbbc0b DBZ-709 PostgreSQL uses scale for time precision not length 2018-06-05 15:51:35 +02:00
luobo
5b33d81f00 DBZ-191 support default value of column in mysql 2018-06-04 10:43:53 +02:00
Gunnar Morling
61f576bb90 DBZ-706 Misc. clean-up 2018-06-01 11:23:30 +02:00
Stephen Powis
40d872f48c DBZ-706 General cleanup 2018-06-01 11:23:30 +02:00
Stephen Powis
867202e0ca DBZ-706 Slight refactoring / shuffling of code to validate source query field in snapshot tests 2018-06-01 11:23:12 +02:00
Jiri Pechanec
764a1d0d58 DBZ-693 SQL parsing in separate method 2018-05-28 12:35:47 +02:00
Jiri Pechanec
45bc7b1536 DBZ-693 On connect statements supported for PostgreSQL 2018-05-28 12:35:47 +02:00
Jiri Pechanec
f74a9fe059 DBZ-693 Initial SQL statements 2018-05-28 12:35:47 +02:00
Jiri Pechanec
f844054f50 DBZ-696 Support nanoseconds in timestamp data 2018-05-22 11:33:50 +02:00
Jiri Pechanec
09b28c0913 DBZ-694 Support milliseconds for pre-epoch dates 2018-05-22 11:22:06 +02:00
Rao
15a90548fb DBZ-666 Supporting ordered snapshot using tables.whitelist config 2018-05-17 06:23:31 +02:00
Jiri Pechanec
0f0a5d4cd4 DBZ-687 Kafka 1.1.0 2018-05-14 09:37:33 +02:00
Omar Al-Safi
8ef9267808 DBZ-668 Added unit test and changed the key field name 2018-03-23 10:50:42 +01:00
Omar Al-Safi
21d2e0b8a9 DBZ-668 Changed the key schema for the heartebat messages from STRING to STRUCT 2018-03-23 10:50:42 +01:00
Jiri Pechanec
2b5370de1a DBZ-667 Sceintific numbers parsing 2018-03-20 14:42:55 +01:00
Gunnar Morling
00b63bb808 DBZ-669 Closing AdminClient instance after usage 2018-03-20 13:07:42 +01:00
Jenkins user
f4e151b23a [maven-release-plugin] prepare for next development iteration 2018-03-20 08:14:19 +00:00
Jenkins user
93b3252332 [maven-release-plugin] prepare release v0.7.5 2018-03-20 08:14:19 +00:00
Gunnar Morling
5f61cb167b DBZ-655 Using AssertJ asserts in UnwrapFromEnvelopTest 2018-03-19 09:28:30 +01:00
Gunnar Morling
5358167446 DBZ-655 Adding test 2018-03-19 09:28:30 +01:00
Maciej Bryński
d5abb8415c DBZ-655 Use cache to avoid using regexp replace 2018-03-19 09:28:23 +01:00
Gunnar Morling
d73c27d9bd DBZ-657 Adding test 2018-03-18 18:26:27 +01:00
Jiri Pechanec
87bd3e9c2f DBZ-657 Handle JSON with control chars not according to specs 2018-03-18 18:03:56 +01:00
Jiri Pechanec
9884a5ca1d DBZ-663 disable cleaning of history topic based on size 2018-03-16 16:14:12 +01:00
Jiri Pechanec
62c7fe12f9 Do not log FLUSH RELAY LOGS statements 2018-03-16 09:51:48 +01:00
Jenkins user
daf27207be [maven-release-plugin] prepare for next development iteration 2018-03-07 08:31:07 +00:00
Jenkins user
9c73774928 [maven-release-plugin] prepare release v0.7.4 2018-03-07 08:31:07 +00:00
Jiri Pechanec
327085164e DBZ-611 Special values supported for double 2018-03-06 16:55:24 +01:00
Gunnar Morling
1f385c21f9 DBZ-611 Typo fix and formatting 2018-03-06 16:55:24 +01:00
Jiri Pechanec
424aefbbb4 DBZ-611 Decimal-> String supported for MySQL 2018-03-06 16:55:24 +01:00
Gunnar Morling
401cbae852 DBZ-632 Removing methods from HistoryRecord which where only used in tests; 
Also adding a new test for HistoryRecord (de-)serialization
 2018-03-02 05:28:16 +01:00
Gunnar Morling
d040ec4427 DBZ-351, DBZ-606 Misc. clean-up; 
* Renaming DebeziumDecimal to SpecialValueDecimal
* Simplifying logic in PostgresValueConverter
 2018-02-27 16:36:21 +01:00
Jiri Pechanec
2534696b21 DBZ-606 Exception thrown for special values when mode handling is not string 2018-02-27 16:36:21 +01:00
Jiri Pechanec
5827a7ac1b DBZ-611 Decimals can be encoded as strings 2018-02-27 16:36:21 +01:00
Jiri Pechanec
c71707afc9 DBZ-351 wal2json usses unlimited precision too 2018-02-27 16:36:21 +01:00
Jiri Pechanec
0c0ac72568 DBZ-606 Type for handling special BigDecimals 2018-02-27 16:36:21 +01:00
Jiri Pechanec
ceaa7f8efe DBZ-351 Numeric is passed with unlimited precision 2018-02-27 16:36:21 +01:00
Gunnar Morling
d89f4e91c6 DBZ-632 Removing unused method parameter 2018-02-23 21:54:41 +01:00
Gunnar Morling
eef1ad7c2a DBZ-630 Refactoring around MongoDB task context; 
* Renaming ConnectorTaskContext to CdcSourceTaskContext
* Renaming ReplicationContext to MongoDbTaskContext
* Making relationship from MongoDbTaskContext to ConnectionContext has-a instead of is-a
 2018-02-21 12:04:41 +01:00
Gunnar Morling
bb12e521f8 DBZ-630 Pulling up getClock() to ConnectorTaskContext 2018-02-21 12:04:41 +01:00
Gunnar Morling
8264c20cf6 DBZ-630 Unifying common start-up logic across connectors 2018-02-21 12:04:41 +01:00
Gunnar Morling
7fe45db831 DBZ-626 Renaming AvroValidator to SchemaNameAdjuster; 
Making apparent that this functionality doesn't solely validate but returns a new name actually
 2018-02-20 09:26:47 +01:00
Gunnar Morling
cddd5fcc8a DBZ-626 Passing schema validator as AvroValidator instance instead of Function 2018-02-20 09:26:47 +01:00
Jiri Pechanec
6e00924ee9 DBZ-625 Fix parsing of float without decimals 2018-02-20 08:47:43 +01:00
Gunnar Morling
b5856f37c7 DBZ-628 Unrelated typo fix 2018-02-20 06:34:42 +01:00
Gunnar Morling
8369f10d52 DBZ-628 Adding MAX_QUEUE_SIZE, MAX_BATCH_SIZE and POLL_INTERVAL_MS to CommonConnectorConfig; 
Also using these options in PG connector
 2018-02-20 06:34:42 +01:00
Gunnar Morling
5bc62dc862 DBZ-620 Removing method only used in tests 2018-02-19 12:18:37 +01:00
Gunnar Morling
ab11f48843 DBZ-620 Creating envelope schemas only once per table type 2018-02-19 12:18:37 +01:00
Jenkins user
6d0cd88e12 [maven-release-plugin] prepare for next development iteration 2018-02-15 04:15:34 +00:00
Jenkins user
7d1e1a989e [maven-release-plugin] prepare release v0.7.3 2018-02-15 04:15:34 +00:00
Gunnar Morling
b1362f9db9 DBZ-278 Extracting broker config retrieval into its own method 2018-02-13 17:12:11 +01:00
Jiri Pechanec
90e4aa0dea DBZ-278 Fail if topic is not created 2018-02-13 16:48:42 +01:00
Jiri Pechanec
4541fc53c9 DBZ-278 Explicit history storage initialization 2018-02-13 16:48:42 +01:00
Jiri Pechanec
ad7b92310e DBZ-278 Test for broker without auto-created topics 2018-02-13 16:48:42 +01:00
Jiri Pechanec
55e81c458b DBZ-278 Create database history topic 2018-02-13 16:48:42 +01:00
Jiri Pechanec
5a0c7c07b0 DBZ-609 DBZ-571 Refactor Postgres type handling 
TypeRegistry introduced for Postgres connector

JDBC column does not have a special componentType

JDBC column provide a database specific type id

OID is a primary type identifier to be used in Postgres connector code - dropping JDB/OID dichotomy
 2018-02-13 14:54:42 +01:00
Jiri Pechanec
dbac1429b2 DBZ-581 Improve error handling for replicators 2018-02-13 14:40:54 +01:00
Jiri Pechanec
ecdce6529d DBZ-220 Null Heartbeat object. Messages contains key 2018-02-13 12:26:48 +01:00
Jiri Pechanec
682cabb84a DBZ-220 Generate heartbeat events in binlog thread 2018-02-13 12:24:51 +01:00
Gunnar Morling
96b4023839 DBZ-220 Misc. improvements; 
* interpreting heartbeat interval as MS
* adding Configuration#getDuration()
 2018-02-13 12:24:51 +01:00
Jiri Pechanec
3c1ea8206b DBZ-220 Refactor heartbeat to a separate class 2018-02-13 12:24:51 +01:00
Gunnar Morling
2e0b9e5e33 DBZ-582 Moving TOMBSTONES_ON_DELETE to CommonsConnectorConfig for re-use 2018-02-13 10:19:18 +01:00
Jiri Pechanec
d0bd1560ff DBZ-576 Make DDL filter internal parameter 2018-02-12 11:22:28 +01:00
Jiri Pechanec
3740aee4e8 DBZ-576 Internal parameters 2018-02-12 11:22:28 +01:00
Gunnar Morling
a77c8d733a DBZ-580, DBZ-586 Making enqueue() and thus snapshots in Postgres interruptable; 
* ChangeEventQueue#enqueue() checks the interrupted state of the calling
thread now, raising an InterruptedException in case the interrupted flag
has been set (because the producer's thread executor has been stopped)
* RecordSnapshotProducer has been adjusted to check for the interrupted
regularly, aborting if it has been set
 2018-02-11 10:05:06 +01:00
Gunnar Morling
45a9847d42 DBZ-580 Initial implementation of dedicated change message queue: 
* To be used across connectors, unifying the enqueuing/polling logic and handling of related config options
* Using it for Postgres connector
 2018-02-11 10:05:06 +01:00
Gunnar Morling
2a724d9611 DBZ-537 Misc. adjustments: 
* Renaming ConfigurationHelper to Instantiator
* Doc improvements and typo fixes
* Bringing getInstance() methods into consistent order
* Raising exception instead of logging error if instantion fails
 2018-02-09 15:45:57 +01:00
Jiri Pechanec
5be55ae8ff DBZ-537 OffsetCommitPolicy supports Configuration constructor initialization 2018-02-09 15:19:48 +01:00
Gunnar Morling
7b6db00db7 DBZ-593 JavaDoc 2018-02-02 13:16:43 +01:00
Gunnar Morling
712824e43f DBZ-593 Passing version explicitly from SourceInfos to base class instead of relying specific packaging structure 2018-02-02 13:16:43 +01:00
Jiri Pechanec
7ebee94169 DBZ-593 Add Debezium version to source in envelope 2018-02-02 13:16:43 +01:00
Gunnar Morling
3a281d1185 DBZ-595 Making sure resources are cleaned up when snapshotting fails; 
* shutting down the snapshotting thread and the DB history producer client
  if the connector is stopped while trying to write to the history topic
* reducing the time that KafkaProducer#send() will block if Kafka isn't
  available; this will release the producer thread quicker in case the
  connector is stopped during snapshotting
* not returning from finally block (!) in case the TX is rolled back; This
  prevented the failed state to be set by the outer catch clause in execute()
 2018-02-02 09:06:32 +01:00
Jiri Pechanec
92740a3626 DBZ-587 Fix thread leak, thread names 2018-02-01 10:04:20 +01:00
Jiri Pechanec
9b592204ac DBZ-587 Centralize and unify thread management 2018-02-01 10:04:20 +01:00
Jenkins user
04624341f5 [maven-release-plugin] prepare for next development iteration 2018-01-25 09:39:44 +00:00
Jenkins user
898f6884e1 [maven-release-plugin] prepare release v0.7.2 2018-01-25 09:39:44 +00:00
Gunnar Morling
8edcf9f3d8 DBZ-507 Making "wkb" optional in Point schema to keep compatability with previous version 2018-01-24 16:51:54 +01:00
Robert Coup
f9d90a482f DBZ-507 Expanding support for geometry types; 
* Adding support for PostGIS geometry types
* Adding support for GEOMETRY, POLYGON and more in MySQL
* For all newly supported types, changes are represented using two new schema types Geometry and Geography, containing the WKB (binary geo data) ans srid (coord system identifier)
* The existing Point type also contains the new (optional) srid field
 2018-01-24 16:51:49 +01:00
Jiri Pechanec
01577b40c3 DBZ-541 Only whitelisted tables are recorded in history 2018-01-24 14:43:57 +01:00
Gunnar Morling
ea4b366b7f DBZ-538 Capturing unparseable DDL for logging in one more place 2018-01-24 11:30:51 +01:00
Peter Goransson
7d78357087 DBZ-443 Adding db-history recovery mode: schema_only_recovery 2018-01-23 09:17:39 +01:00
Gunnar Morling
0c4190c493 DBZ-516 Using Duration instead of long in a few more places 2018-01-18 14:13:58 +01:00
Jiri Pechanec
24bdcaf059 DBZ-516 Return control to Connect periodically 2018-01-18 14:13:58 +01:00
Gunnar Morling
9f433b1e24 DBZ-544 Comment fix 2018-01-17 15:16:37 +01:00
Jiri Pechanec
cdf44faba4 DBZ-544 Log matching DDL filter 2018-01-17 15:16:37 +01:00
Gunnar Morling
4d1d016acb DBZ-392 Making table map and set dedicated classes just with the needed API 2018-01-17 11:21:02 +01:00
Jiri Pechanec
9f0f0fe5d4 DBZ-392 Support for MySQL running on lowercase filesystems 2018-01-17 11:21:02 +01:00
Gunnar Morling
b99bdf7fdc DBZ-543 Removing a few unused methods 2018-01-15 14:37:46 +01:00
Gunnar Morling
5c88431c07 DBZ-494 Making tests more lenient towards specific List implementations; 
also fixing a few typos.
 2018-01-15 10:40:50 +01:00
Tom Bentley
5b839d3665 DBZ-540 Adding KafkaCluster#zkPort() 
If you start a cluster (e.g. in a test) without specifying a port
you get a random port. Sometimes you might want to connect to the
embedded zookeeper instance (for instance, to make an assertion about
a znode). To do this you need to know the port number. So let's expose it.
 2018-01-12 18:25:41 +01:00
Jiri Pechanec
0b269a6e41 DBZ-538 Improve invalid DDL statement reporting 2018-01-12 12:26:30 +01:00
Jiri Pechanec
2a7377a833 DBZ-455 Use valueOf instead of constructors 2018-01-05 02:32:37 +01:00
Jenkins user
6bb34b42f9 [maven-release-plugin] prepare for next development iteration 2017-12-20 07:15:12 +00:00
Jenkins user
16dcd4c980 [maven-release-plugin] prepare release v0.7.1 2017-12-20 07:15:12 +00:00
Jenkins user
5e09932cb9 [maven-release-plugin] prepare for next development iteration 2017-12-15 05:10:23 +00:00
Jenkins user
6c1d61e03b [maven-release-plugin] prepare release v0.7.0 2017-12-15 05:10:23 +00:00
Gunnar Morling
4e8cedd094 DBZ-379 Postgres connector minimizes use of JDBC metadata 2017-12-13 12:20:37 +01:00
Jiri Pechanec
5ae236241b DBZ-469 Fixed a backslash regex in testcase 2017-12-11 16:58:23 +01:00
Gunnar Morling
8e99dc3abd DBZ-469 Adding one more test case 2017-12-11 16:58:23 +01:00
Gunnar Morling
1c55e41941 DBZ-469 Renaming listOfRegex() to setOfRegex(); 
* Simplifying test of that method
* Adding test to ensure correctness of default DDL pattern
 2017-12-11 16:58:23 +01:00
Jiri Pechanec
86d9e109fc DBZ-469 Filter our RDS heartbeat INSERT statements 2017-12-11 16:58:23 +01:00
rkerner
c7ac481c43 [DBZ-342] fix broken MySQL data type "TIME" handling 2017-11-29 20:34:12 +01:00
Gunnar Morling
a55227aa83 DBZ-464 Don't stop after reaching max retry count, but raise an exception instead; 
Also increasing default value, as the connector can't start its work without fully recovering the history
 2017-11-28 08:47:27 +01:00
Gunnar Morling
6537d904ce DBZ-464 Reading until end offset of history topic 2017-11-28 08:47:27 +01:00
Gunnar Morling
bc2d0e5956 DBZ-464 Removing unused method parameters from AbstractDatabaseHistory#recoverRecords() 2017-11-28 08:47:27 +01:00
Jiri Pechanec
20a2cdfdea DBZ-476 Doubled quotes are parsed as escaped 2017-11-23 14:51:51 +01:00
Jiri Pechanec
57e7f84163 DBZ-479 Forced fsync slows down tests 2017-11-22 14:34:59 +01:00
Gunnar Morling
2b3276be1d DBZ-478 Correctly handling null value converters; 
* using simple for loop for the sake of easier debugging
* log info about unsupported column type on WARN rather than TRACE level
 2017-11-22 09:43:10 +01:00
David Szabo
1c07ff4775 DBZ-466 Remove hardcoding of schema version number, leaving it empty instead 2017-11-16 17:43:14 +01:00
Jiri Pechanec
4d253d2987 DBZ-437 String tokens and SQL words are handled separately in procedure parsing 2017-11-14 09:40:54 +01:00
Jiri Pechanec
5e7714eb00 DBZ-456 Reduce text execution time 2017-11-14 06:47:32 +01:00
Henryk Konsek
be40502cc7 DBZ-323 Cluster and server properties should be added into config, not set as default ones 2017-11-13 16:35:57 +01:00
Gunnar Morling
2744f54e9d DBZ-438 Renaming BufferedBlockingConsumer#flush() to close() to make clear it's onyl meant to be called once at the end 2017-11-13 14:19:39 +01:00
Gunnar Morling
2b62943e4f DBZ-438 Avoiding issues due to concurrent usage of BufferedBlockingConsumer 2017-11-13 14:19:39 +01:00
Gunnar Morling
5fbe742be8 DBZ-285 Specifying scope of dependencies in the individual POMs for the sake of comprehensibility 2017-11-10 16:48:32 +01:00
Gunnar Morling
580647b226 DBZ-285 Making more dependencies "provided" 2017-11-10 16:33:02 +01:00
Ewen Cheslack-Postava
8826669b43 DBZ-285: Use provided or test dependencies for Connect and Kafka dependencies 2017-11-04 12:01:24 -07:00
Jiri Pechanec
a6bd883857 DBZ-432 Rebased to Kafka 1.0.0 2017-11-03 11:06:18 +01:00
Jiri Pechanec
e841cbd609 DBZ-429 Fixed signed default value handling 2017-11-03 06:13:05 +01:00
Gunnar Morling
3f410c20e5 DBZ-422 Using existing constants BigDecimal.ONE and ZERO; 
Also using valueOf() methods to benefit from internal caching for most numeric types
 2017-11-02 12:23:06 +01:00
pmx
3c91ad47ec DBZ-422 Extracting numeric constants and boolean conversions into NumberConversions class 2017-11-02 12:12:54 +01:00
pmx
c3ce7c571a DBZ-422 Making sure convertSmallInt() always returns a short 2017-11-02 12:12:54 +01:00
Jiri Pechanec
f93e1e9bcd DBZ-395 Make temp table regex stricter 2017-10-26 11:11:22 +02:00
Jiri Pechanec
1c6e652c71 DBZ-395 Filter out DROP TEMPORARY TABLE statements 2017-10-26 11:11:22 +02:00
Ben Williams
a3b4fedd5f DBZ-363 Add support for BIGINT UNSIGNED handling for MySQL 2017-10-18 10:20:03 +02:00
Gunnar Morling
73189892b3 DBZ-258 Misc. improvements while reviewing the change; 
* Removing superfluous config option
* Making loggers static
* JavaDoc fixes
* Extracting hexStringToByteArray() to helper and adding test
* Removing superfluous super() invocations
 2017-10-18 09:21:22 +02:00
Jiri Pechanec
e47b4cb81c DBZ-258 Changes after first review 2017-10-18 09:21:22 +02:00
Jiri Pechanec
0bc8129961 DBZ-258 Support for wal2json plugin 2017-10-18 09:21:22 +02:00
Jenkins user
75937711fa [maven-release-plugin] prepare for next development iteration 2017-09-21 04:42:02 +00:00
Jenkins user
a89b9332e4 [maven-release-plugin] prepare release v0.6.0 2017-09-21 04:42:02 +00:00
Gunnar Morling
7f20aca03d DBZ-226 Minor wording and formatting improvements 2017-09-20 15:11:13 +02:00
Jiri Pechanec
0afb687303 DBZ-226 SMT to change CDC to simple record 2017-09-20 12:05:12 +02:00
Jiri Pechanec
ba3d7d762b DBZ-318 Support for a Decimale with variable scale schema 2017-09-19 08:39:07 +02:00
Gunnar Morling
99f39038bb DBZ-347 Formatting 2017-09-18 17:04:14 +02:00
Jiri Pechanec
61ccefcc5c DBZ-347 Configurable behaviour for unknown DDL, failing by default 2017-09-18 17:02:21 +02:00
Jiri Pechanec
f7736cd0ab DBZ-341 Ignore malformed messages coming from database history 2017-09-18 17:02:21 +02:00
Jiri Pechanec
01980cf29b DBZ-341 Ignore malformed messages coming from database history 2017-09-15 11:52:18 +02:00
Jiri Pechanec
b53b4f67d2 DBZ-319 Views are now ignored in snapshots 2017-09-15 11:37:38 +02:00
Jiri Pechanec
e4bc6670c8 DBZ-305 Rebase build process against Kafka 0.11 2017-08-17 18:51:01 +02:00
Jenkins user
214696ef0c [maven-release-plugin] prepare for next development iteration 2017-08-17 11:51:05 +00:00
Jenkins user
c867e6fea6 [maven-release-plugin] prepare release v0.5.2 2017-08-17 11:51:05 +00:00
Gunnar Morling
1665026c8a DBZ-298 Adding support for quoted identifiers for schemas, tables and 
columns; expanding test
 2017-08-17 09:08:17 +02:00
Gunnar Morling
564d90dc09 DBZ-298 Formatting 2017-08-17 09:03:16 +02:00
Gunnar Morling
a4dc7620fe DBZ-327 Simplifying validation implementation a bit via Strings#isNullOrEmpty() 2017-08-16 14:37:01 +02:00
Mario Mueller
2a2f911e74 DBZ-327 Fix broken SMT validation and add some tests for the 
configuration combinations
 2017-08-16 11:57:48 +02:00
Emrul Islam
bb79b2f60f DBZ-297 Initial support for some Postgres array types (one-dimensional only) 2017-07-27 10:04:13 +02:00
Gunnar Morling
79fbc028a8 DBZ-311 Precompiling and simplifying some regular expressions 2017-07-25 21:43:26 +02:00
Gunnar Morling
825530256e DBZ-311 Removing trailing whitespace 2017-07-25 21:43:26 +02:00
Gunnar Morling
a8d1817c22 [maven-release-plugin] prepare for next development iteration 2017-06-09 16:14:31 +00:00
Gunnar Morling
3f512aace7 [maven-release-plugin] prepare release v0.5.1 2017-06-09 16:14:31 +00:00
Gunnar Morling
2a22f3d4b0 DBZ-254 Right-padding values for fixed-length BINARY columns with 0x00 (zero byte) characters for MySQL; 
Also fixing JDBC types for binary data types for MySQL.
 2017-05-30 15:04:38 +02:00
Tom Bentley
96eda35da5 Respect any existing auto.create.topics.enable when creating a server 2017-05-23 16:58:12 +01:00
dleibovic
a683629519 Change ByLogicalTableRouter logging from info to debug 
This can be a bit spammy in the info level
 2017-05-19 12:09:40 -04:00
Randall Hauch
02e655fa53 DBZ-198 Add another DDL parser test case 
Added an additional test that is unable to reproduce the problem reported on April 4.
 2017-05-19 15:50:03 +02:00
Jiri Pechanec
ec915c559c [DBZ-246] Correctly handle enumeratedvale type 2017-05-09 11:24:28 +02:00
Gunnar Morling
1158833524 DBZ-238 Removing some trailing whitespace 2017-05-05 21:59:06 +02:00
Gunnar Morling
64958d11db DBZ-238 Moving EnumeratedValue to its own file 2017-05-05 21:55:46 +02:00
Brendan Maguire
928c9cb5f0 DBZ-238: Fix Postgres database.sslmode = "require" bug 
* Also added EnumeratedValue interface for Configuration so getValue is used instead of toString when getting the config value
 2017-05-05 21:55:45 +02:00
Gunnar Morling
1074de4efa DBZ-222 Some more clean-up 2017-05-04 09:25:25 +02:00
Omar Al-Safi
ea092f1baf DBZ-222 Add null to Point comment 2017-05-04 08:53:46 +02:00
Gunnar Morling
5630b61be6 DBZ-222 dependency clean-up 2017-05-04 08:53:05 +02:00
Omar Al-Safi
791545c5f4 DBZ-222 Added support for MySQL POINT type 2017-05-04 08:53:05 +02:00
Randall Hauch
d42736e065 DBZ-121 Corrected JavaDoc and reorganized imports 2017-04-04 16:53:35 -05:00
dleibovic
8e304c4cba rename fields and simplify descriptions 2017-04-03 11:35:33 -04:00
dleibovic
fa157b0159 make the name of the physical table identifier field added to the key schema configurable 2017-03-31 13:09:32 -04:00
dleibovic
d7acda04c4 DBZ-121: routing to custom topic names via Kafka Connect Transformations. 2017-03-30 15:02:59 -04:00
Randall Hauch
709cd8f3fe [maven-release-plugin] prepare for next development iteration 2017-03-27 11:28:12 -05:00
Randall Hauch
2bc3d45954 [maven-release-plugin] prepare release v0.5.0 2017-03-27 11:28:11 -05:00
Randall Hauch
81f62b6961 DBZ-205 Corrected MySQL connector to handle 2-digit years 
MySQL has special handling of 2-digit years that it deems are ambiguous, such as the year value `17` that is actually treated as `2017`. Apparently the 2-digit values are stored in MySQL and the interpretation is performed when the data is extracted, so therefore the connector needs to also perform this adjustment of the year values. This commit uses the JDK’s `TemporalAdjuster` interface and passes this down to the requisite temporal-related datatype handling code. The MySQL connector then provides its own `TemporalAdjuster` implementation that adjusts the year values via the excellend JDK `Temporal` methods.

A row in one of the MySQL test databases was changed to use a 2-digit year of `16` while the test method still checks that the year is still 2016`, verifying that the year value is properly adjusted.
 2017-03-27 10:58:21 -05:00
Randall Hauch
613e6b6340 Fixed compilation warnings 2017-03-27 10:37:15 -05:00
Randall Hauch
7a72ed6ae6 Merge pull request #202 from don41382/upgrade-kafka-version-to-0.10.2.0 
DBZ-203 Upgrade kafka version from 0.10.1.1 to 0.10.2.0
 2017-03-17 17:21:48 -05:00
Randall Hauch
430d756062 [maven-release-plugin] prepare for next development iteration 2017-03-17 15:41:58 -05:00
Randall Hauch
536cbf6300 [maven-release-plugin] prepare release v0.4.1 2017-03-17 15:41:57 -05:00
Randall Hauch
e5ee3847dd DBZ-195 Added tests to try to replicate a reported issue 
Added a table and inserted rows that tries to replicate the problem reported in DBZ-195, but the test was unable to replicate the problem. In fact, this really is no different than existing tests. Changed the log messages so that if/when this happens again it will be possible to know which row was problematic.
 2017-03-17 10:47:32 -05:00
Felix Eckhardt
5d414c521a upgraded kafka version from 0.10.1.1 to 0.10.2.0 2017-03-17 11:36:15 +11:00
Randall Hauch
cf5391482a DBZ-198 Improved MySQL DDL parser to better handle blocks 
The MySQL parser now properly handles control blocks such as `BEGIN…END`, `IF…END IF`, `REPEAT…END REPEAT`, and `LOOP…END LOOP`, even in cases where the block is preceded by and terminated by a label.
 2017-03-16 13:32:21 -05:00
Randall Hauch
b48ccce4b5 DBZ-200 Corrected MySQL DDL parser to better handle column definitions 
Apparently not all reserved words must be quoted when using them as colum names, so refactored MySQL’s DDL parser to better handle a variety of unquoted colum names that are reserved words.
 2017-03-08 12:12:27 -06:00
Randall Hauch
d2986710a5 DBZ-188 More efficient GTID source filters for MySQL Connector 
Changed the GTID source filters in the MySQL connector to be far more efficient when the filters specify literal UUIDs rather than regex patterns. In these cases, the predicate just checks whether a supplied value is in a hash set, and no regular expression patterns are used.

The GTID source filters can still be a combination of UUID literals and regular expressions, and the predicate will use the best implementation for each. For example, if the filters include all UUID literals, then regular expressions will never be used.
 2017-02-10 11:34:24 -06:00
Randall Hauch
8c60c29883 [maven-release-plugin] prepare for next development iteration 2017-02-07 14:22:12 -06:00
Randall Hauch
20134286e9 [maven-release-plugin] prepare release v0.4.0 2017-02-07 14:22:11 -06:00
Randall Hauch
74e5ba6448 DBZ-176 Corrected MySQL DDL parser to support creating triggers with definers 
The MySQL DDL parser was not correclty handling `DEFINER` clauses within `CREATE TRIGGER` or `CREATE EVENT` statements. Support for `DEFINER` clauses was recently added for the various forms of `CREATE PROCEDURE`, `CREATE FUNCTION` and `CREATE VIEW` statements. These are the only kinds of statements that have the definer attribute, per the [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html).
 2017-02-02 12:44:28 -06:00
Randall Hauch
972cfbe2c4 DBZ-173 Additional fixes to KafkaDatabaseHistory class for Kafka 0.10.1.0 
The KafkaDatabaseHistory class was not behaving well in tests using my local development environment. When restoring from the persisted Kafka topic, the class would set up a Kafka consumer and see repeated messages. It is unclear whether the repeats were due to our test environment and very short poll timeouts. Regardless, the restore logic was refactored to track offsets so as to only process messages once.
 2017-02-01 14:47:41 -06:00
Horia Chiorean
d035c4bc8d DBZ-173 Changes the MySQL ITs to not use TZ information for expected dates and fixes the character set for parsing test files 2017-01-27 14:53:10 +02:00
Horia Chiorean
a2154d3d32 DBZ-173 Changes the MySQL ITs to use the database.hostname system property instead of always hardcoding 'localhost' 2017-01-27 09:19:57 +02:00
Horia Chiorean
7dfdef3558 DBZ-173 Upgrades the Kafka artifact versions to 0.10.1.1 2017-01-27 09:19:57 +02:00
Horia Chiorean
031c4a1552 DBZ-183 Fixes the BinlogReader's handling of TIMESTAMP columns to correctly account for timezones 2017-01-25 16:39:36 +02:00
Randall Hauch
f0db8d1b1f DBZ-179 Corrected JavaDoc in PostgreSQL connector 
Corrected the JavaDoc and removed trailing spaces in the PostgreSQL connector code.
 2017-01-20 11:51:17 -06:00
Randall Hauch
a73f85a80f Merge pull request #162 from rareddy/DBZ-177 
DBZ-177: Providing an alternative way to create JDBC connection based …
 2017-01-13 13:37:38 -06:00
Ramesh Reddy
a9aace3480 DBZ-177: Providing an alternative way to create JDBC connection based on the configured JDBC driver class name and supplied classloader. The loading/creating the JDBC connections is not reliable when driver libraries in a different classloader than the DriverManager. 2017-01-13 12:58:14 -06:00
Horia Chiorean
ae85656851 DBZ-3 Fixes topic naming to include the name of the server 2016-12-30 09:42:48 +02:00
Horia Chiorean
737614a555 DBZ-3 Implements a connector for streaming changes from a Postgres database 
The version of the DB server required for this to work is at least 9.4. To be able to stream logical changes, the code relies on enhancements to the JDBC driver which are not yet public. Therefore, the current codebase includes the sources for the JDBC driver.
The commit also updates the general DBZ build system for:
* custom checkstyle package exclusions - required by the Postgres driver the protobuf code for now
* adds support for debugging Surefire and Failsafe
 2016-12-27 14:44:32 +02:00
Horia Chiorean
23e3f59fa1 DBZ-3 Implements a connector for streaming changes from a Postgres database 
The version of the DB server required for this to work is at least 9.4
The commit also updates the general DBZ build system for:
* custom checkstyle package exclusions - required by the Postgres driver the protobuf code for now
* adds support for debugging Surefire and Failsafe
 2016-12-27 14:44:32 +02:00
Horia Chiorean
8e14f150db DBZ-3 Adds the structure for a Postgres connector which uses a Debezium Postgres docker image that has the decoderbufs plugin enabled to read WAL changes 2016-12-27 14:44:29 +02:00
Randall Hauch
5dceb05f69 DBZ-151 Additional changes to improve test framework and MySQL integration tests 2016-12-20 10:58:56 -06:00
Randall Hauch
a3bece4472 DBZ-151 Added new integration test framework for easily comparing output of connectors to expected results. 2016-12-20 09:18:09 -06:00
Randall Hauch
0851d8280c DBZ-166 Corrected shutdown logic of MySQL connector 
The MySQL connector uses several threads, so previously upon connector shutdown these threads were simply cancelled. This is fine for the binlog reader (which can stop at any moment), but is a poor approach for the snapshot as we didn’t always properly release the database resources and also didn’t complete the writing of the DDL history.

With this change, the snapshot reader stops in a very controlled manner, basically by having the 10-step snapshot procedure frequently check whether the reader is to continue working, and to completely avoid thread interruption altogether. And, the snapshot procedure will always clean up its database resources (locks, transactions, etc.), even if the procedure is stopped before completion.

This change also refactors how the snapshot and binlog reader are managed. This is no longer done in the MySqlConnectorTask class (which is busy enough), but rather the logic has been encapsulated in a new `ChainedReader` that makes use of a new `Reader` interface. This makes testing of `ChainedReader` easier, and ensure that `ChainedReader` relies only upon the primary methods of `Reader` rather than upon `AbstractReader`. `ChainedReader` handles multiple readers generically, and ensures that when stopped the readers are all handled correctly and completely process all records, yet avoid accidentally starting a subsequent reader(s) when stopping the previous reader.
 2016-12-15 10:55:18 -06:00
Randall Hauch
c762a221b7 DBZ-162 Corrected DDL parsing of MySQL functions 
The MySQL DDL parser was not properly consuming function declarations. For functions, the parser consumes the entire statement without handline the various expressions within the function declaration, but the parser was not properly finding the end of the statement and instead was continuing to try to consume values beyond the end of the statement.

Specifically, when the parser consumes a `BEGIN`, it looks for a corresponding `END`. However, if it encountered an `END IF`, the `IF` plus any remaining tokens were left on the token stream and unprocessed. This confused the parser, which keep looking for statements and ultimately ended with a `No more content` error.

This case was replicated in integration tests, and the code fixed to properly find the end of the statements.
 2016-12-06 17:34:52 -06:00
Sherafudheen PM
ee52219736 DBZ-160 - Issue while parsing create table script with ENUM type and default value 'b' 2016-12-02 17:42:44 +05:30
Randall Hauch
d80bc1bfd7 DBZ-153 MySQL connector supports enum and set values with parentheses 
Changed the MySQL connector to support ENUM and SET literals with parentheses.
 2016-11-14 12:22:08 -06:00
Randall Hauch
b0ded5f383 DBZ-147 Added ability to treat MySQL DECIMAL as double 
By default the MySQL connector handles `DECIMAL` and `NUMERIC` columns using `java.math.BigDecimal` values and describing them using the `org.apache.kafka.connect.data.Decimal` schema type, which serializes the values to a binary form.

This change adds a configuration option that will keep the default behavior, but will instead allow handling `DECIMAL` adn `NUMERIC` values as Java `double` and a schema type of `FLOAT64`.
 2016-11-09 11:27:09 -06:00
Randall Hauch
ea5f7983c7 DBZ-144 Corrected MySQL connector restart 
Added tests to verify whether the connector is properly restarting in the binlog when previously the connector failed or stopped in the middle of a transaction. The tests showed that the connector is not able to properly start when using or not using GTIDs, since restarting from an arbitrary binlog event causes problems since the TABLE_MAP events for the affected tables are skipped.

The logic was changed significantly to record in the offsets the binlog coordinates at the start of the transaction, which should work whether or not GTIDs are used. Upon restart, the connector may have to re-read the events that were previously processed, but now the offset also includes the number of events that were previously processed so that these can be skipped upon restart.

This has an unforunate side effect since the offsets capture a transaction was completed only when it generates a source record for the subsequent transaction. This is because the connector generates source records (with their offsets) for the binlog events in the transaction before the transaction's commit is seen. And, since no additional source records are produced for the transaction commit, the recorded offsets will show that the prior transaction is complete and that all of the events in the subsequent transaction are to be skipped. Thus, upon restart the connector has to re-read (but ignore) all of the binlog events associated with the completed transaction. This shouldn’t be a problem, and will only slow restarts for very large transactions.
 2016-11-09 08:11:41 -06:00
Randall Hauch
207315e5df DBZ-146 Improved error handling of MySQL Connector 
Improved the error handling of the MySQL connector to ensure that we’re always stopping the connector when we have a problem handling a binlog event or if we have problems starting.
 2016-11-03 16:55:59 -05:00