alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
5,302 Commits 3 Branches 0 Tags 51 MiB
224463a57b
Commit Graph

1103 Commits

Author SHA1 Message Date
jcechace
d9657dab1a [DBZ-1273] Bumping JDBC and Mongo driver versions 2019-05-18 05:50:27 +02:00
Gunnar Morling
c7d2a35a0e DBZ-1264 Member ordering 2019-05-18 05:48:54 +02:00
Jiri Pechanec
a8f6533c66 DBZ-1264 Ignore unique index for non-monitored tables 2019-05-18 05:46:56 +02:00
Chris Cranford
f67ce1b8be DBZ-733 Consolidate parser tests 2019-05-17 13:18:24 -04:00
Chris Cranford
efa22016a8 DBZ-736 DBZ-733 Removed MySQL legacy parser and consolidate parser tests 2019-05-17 13:18:24 -04:00
Chris Cranford
373e508d70 DBZ-736 Remove MySQL legacy parser 2019-05-17 13:18:24 -04:00
Jiri Pechanec
13e5fe4537 DBZ-1228 Simplify error handling 2019-05-17 10:24:02 -04:00
Arkoprabho Chakraborti
ceb56c1c57 DBZ-1228 Changes as suggested in PR #867 
Used multiple catch statements and handle in 1 branch.

Not using static modifier for keymanager.
 2019-05-17 10:24:02 -04:00
Arkoprabho Chakraborti
b7526d537d DBZ-1228 MySQL connection with client authentication does not work 
The SSL properties were being reset, and thus the authentication was failing on the second time.
When debezium connects to the database the first time in order to test the connections, the required parameters are all set. And thus the test passes.
After that the shutdown method is called that clears these properties, and the consecutive connection fails.

DBZ-1228: MySQL connection with client authentication does not work

Create and pass a socket factory that provides they keystore config

DBZ-1228: MySQL connection with client authentication does not work

Prevent forcing of SSL. Use SSL only if the required properties are set.

DBZ-1228: MySQL connection with client authentication does not work

Added better exception messages
 2019-05-17 10:24:02 -04:00
Jenkins user
7748be6056 [maven-release-plugin] prepare for next development iteration 2019-05-02 16:04:56 +00:00
Jenkins user
5af083b86e [maven-release-plugin] prepare release v0.9.5.Final 2019-05-02 16:04:55 +00:00
Chris Cranford
9404eff794 DBZ-1226 Cleaned up tests 2019-05-01 18:37:09 -04:00
Gunnar Morling
2cd0eb17db DBZ-739 Removing obsolete method 2019-05-01 23:17:00 +02:00
Chris Cranford
46111505ed DBZ-1226 Fix test to validate enum/set options directly from antlr parsing. 2019-05-01 21:17:42 +02:00
Chris Cranford
a5fdfd9b24 DBZ-1226 Replace double single-quotes with a singular quote. 2019-05-01 21:17:42 +02:00
Chris Cranford
c6aa02f813 DBZ-1226 Add support for enum/set values that contain commas or escaped characters. 2019-05-01 21:17:42 +02:00
jchipmunk
72b06366fe Removing extra initialisation of "config" variable in PG config 2019-04-30 09:57:11 +02:00
jchipmunk
9fe033ce24 DBZ-1247 Specifying default snapshot fetch size for each connector config 2019-04-30 09:57:11 +02:00
Chris Cranford
83bf0f861a DBZ-1243 Fix handling of default value for datetime(0) 2019-04-25 17:31:22 +02:00
Jiri Pechanec
79afafcb0d DBZ-1249 Fix default value conversion for larger bits 2019-04-23 13:02:31 +02:00
krizhan
70c564b4c2 DBZ-212 Moved SERVER_NAME to RelationalDatabaseConnectorConfig 2019-04-23 12:29:44 +02:00
Gunnar Morling
5339dfc289 DBZ-1233 Formatting 2019-04-23 12:18:53 +02:00
Mathieu Rozieres
df79dfe751 DBZ-1233 MODE and others as MySQL column names 
Add MODE to keywordsCanBeId to fix parse exception on so named columns
 2019-04-23 12:18:40 +02:00
Jenkins user
cefc5ccfb9 [maven-release-plugin] prepare for next development iteration 2019-04-11 12:07:16 +00:00
Jenkins user
fc3026adc3 [maven-release-plugin] prepare release v0.9.4.Final 2019-04-11 12:07:15 +00:00
Gunnar Morling
f04147d67d DBZ-1208 Misc. clean-up; 
* Adding reference to source of logic for setting up SSL socket factory
* Making method side-effect free
* Adding Jordon to COPYRIGHT.txt
 2019-04-10 11:41:51 +02:00
jbragg
9ae2ebe329 DBZ-1208 Support for TLSv1.2; 
Using accepted TLS protocol version  to set for BinaryLogClient
 2019-04-10 11:41:13 +02:00
Jiri Pechanec
50fbe32113 DBZ-1220 Integration test for schema metadata 2019-04-10 11:20:09 +02:00
Jiri Pechanec
394eb74d0f DBZ-1220 FLOAT supports also single precision constraint 2019-04-10 11:20:09 +02:00
Jiri Pechanec
de9e80a58d DBZ-1143 Describe test values; improve tests 2019-04-10 10:59:02 +02:00
Jiri Pechanec
9c7f181174 DBZ-1143 Year adjustment configurable; test 19xx years 2019-04-10 10:59:02 +02:00
Jiri Pechanec
155e63adab DBZ-1143 Change default to Debezium-made conversion 2019-04-10 10:59:02 +02:00
Jiri Pechanec
16e731b748 DBZ-1143 2-digit years are converted by database by default 2019-04-10 10:59:02 +02:00
Steviep
7742e66e11 DBZ-1221 Creating new metric IsGtidModeEnabled 2019-04-09 14:52:18 +02:00
Steviep
94f71ea96e DBZ-1206 Adding new connector metric "numberOfFilteredEvents" 2019-04-09 14:46:54 +02:00
Chris Cranford
664c617661 DBZ-1203 Replace collectionOption rule with STRING_LITERAL token. 2019-04-08 11:16:42 +02:00
Chris Cranford
38e1cc56bb DBZ-1203 Fix parsing alter column with enum and character set 2019-04-08 11:16:42 +02:00
Joy Gao
08271ab444 DBZ-1218 Close JDBC connection in SnapshotReader for MySQL Connector 2019-04-08 08:47:43 +02:00
jchipmunk
7c0ae3ee20 DBZ-1212 SLF4J usage issues 2019-04-04 21:32:12 +02:00
Jiri Pechanec
b90274cc63 DBZ-1194 Convert nulls even for key 2019-03-29 21:20:53 +01:00
Jiri Pechanec
e6b956b36d DBZ-1204 Allow setting boolean as default value for TINYINT 2019-03-29 09:41:44 +01:00
ShubhamRwt
540a951211 DBZ- 362 Adding WhitespaceAfter check to Checkstyle 2019-03-28 09:24:11 +01:00
Jenkins user
8b0edd1871 [maven-release-plugin] prepare for next development iteration 2019-03-25 12:16:31 +00:00
Jenkins user
56f98a48b2 [maven-release-plugin] prepare release v0.9.3.Final 2019-03-25 12:16:30 +00:00
Jiri Pechanec
9b405aa84d DBZ-1039 Add missing spaces 2019-03-25 10:23:14 +01:00
shubham
8cca21e969 DBZ-1039 Make all ifs with braces 2019-03-25 10:23:14 +01:00
Jiri Pechanec
13f49f02a9 DBZ-1185 Use Boolean instead of Optional 2019-03-21 14:02:23 +01:00
Jiri Pechanec
958d207121 DBZ-1185 Skip unsupported tests for legacy parser 2019-03-21 14:02:23 +01:00
Jiri Pechanec
601f2008ac DBZ-1185 SERIAL is valid table/column id 2019-03-21 14:02:23 +01:00
Jiri Pechanec
76e7cd6664 DBZ-1185 Support for SERIAL default value 2019-03-21 14:02:23 +01:00
Jiri Pechanec
4d4741a207 DBZ-1185 Support for SERIAL type alias 2019-03-21 14:02:23 +01:00
Jiri Pechanec
915bdc9b7f DBZ-1186 Comma is not mandatory between table options 2019-03-21 11:22:07 +01:00
Jiri Pechanec
f3c0c94b40 DBZ-1188 Use string testing utility method 2019-03-21 10:38:32 +01:00
Jiri Pechanec
2df2a62ddf DBZ-1188 False pasitive warning for non-empty password 2019-03-21 10:32:06 +01:00
Jiri Pechanec
c740400587 DBZ-1184 Run the test as the last in testsuite 2019-03-18 12:12:12 +01:00
Jiri Pechanec
08bc66f8ef DBZ-1184 Simplify the test 2019-03-18 12:12:12 +01:00
Jiri Pechanec
a892a4eb6f DBZ-1184 Add test 2019-03-18 12:12:12 +01:00
Jiri Pechanec
41734d322e DBZ-1184 Support long purged GTID set 2019-03-18 12:12:12 +01:00
Jenkins user
50a7c568fa [maven-release-plugin] prepare for next development iteration 2019-02-23 09:53:58 +00:00
Jenkins user
28f3839804 [maven-release-plugin] prepare release v0.9.2.Final 2019-02-23 09:53:58 +00:00
Jiri Pechanec
c7e0ff231b DBZ-1156 DBZ-1157 Upgrade to Kafka 2.1.1, Confluent Plat. 5.1.2 2019-02-21 09:39:05 +01:00
Gunnar Morling
510e9415f9 DBZ-1137 Some clean-up; 
* Using existing justifyLeft() method
* Giving expected values as Duration
* Removing unused methods
 2019-02-19 18:41:09 +01:00
Jiri Pechanec
0d60e60243 DBZ-1137 Default for TIME can be generic interval value 2019-02-19 13:40:01 +01:00
Jiri Pechanec
46aad6bcf5 DBZ-1147 Single quotes for string constant 2019-02-18 14:41:03 +01:00
Jiri Pechanec
14c1f30ca3 DBZ-1150 CHECK TABLE options are valid identifiers 2019-02-18 14:38:36 +01:00
Jenkins user
b96e5fee0d [maven-release-plugin] prepare for next development iteration 2019-02-13 11:13:44 +00:00
Jenkins user
3de2d04bcd [maven-release-plugin] prepare release v0.9.1.Final 2019-02-13 11:13:44 +00:00
Gunnar Morling
6a871e1b65 DBZ-983 Expanding test 2019-02-07 15:04:43 +01:00
Jiri Pechanec
eff78cae5a DBZ-983 Handle rounding of defaults 2019-02-07 15:04:43 +01:00
Jiri Pechanec
3487cf1bfb DBZ-983 BigDecimal default value scales according to column 2019-02-07 15:04:43 +01:00
Jiri Pechanec
6637f68861 DBZ-1123 Integration test for generated defaults 2019-02-07 14:06:49 +01:00
Jiri Pechanec
f65cbb7b7d DBZ-1123 Support generated colums 2019-02-07 14:06:49 +01:00
Gunnar Morling
99caad60d8 DBZ-1040 Changing resolution of "lag behind source" to milli-seconds; 
Also using Duration/Instant types internally.
 2019-02-04 12:39:07 +01:00
Gunnar Morling
913e0bc5c8 DBZ-1040 Making metadata provider a member of Metrics instead of passing it for each onEvent() call 2019-02-04 12:39:07 +01:00
Jiri Pechanec
a621e43d8b DBZ-1040 Expose transaction id 2019-02-04 12:39:07 +01:00
Jiri Pechanec
213d4a42ca DBZ-1040 Metrics for internal queue 2019-02-04 12:39:07 +01:00
Jiri Pechanec
d993ee265e DBZ-1040 Metrics for lag behind source, offset and tx count 2019-02-04 12:39:07 +01:00
jchipmunk
ff4e38cc46 DBZ-1112 Strings.join() doesn't apply conversation for first element 
Replace:
- Integer.parseInt() to Integer.valueOf()
- Short.parseShort() and new Short() to Short.valueOf()
- Long.parseLong() to Long.valueOf()
 2019-01-30 09:36:09 +01:00
Gunnar Morling
dd01d26d16 DBZ-1113 Pulling up test to MySqlDdlParserTest; 
* Adding Joy Gao to COPYRIGHT.txt
* Typo fix
 2019-01-30 09:22:05 +01:00
Joy Gao
030cf9ac14 DBZ-1113 allow adding multiple partitions in a single statement 2019-01-30 09:22:05 +01:00
Jenkins user
c5b7d21d3e [maven-release-plugin] prepare for next development iteration 2019-01-28 11:07:04 +00:00
Jenkins user
f1ae79ff73 [maven-release-plugin] prepare release v0.9.0.CR1 2019-01-28 11:07:04 +00:00
Gunnar Morling
ccff79e07d DBZ-1105 Removing TableImpl#columnNames field 2019-01-28 10:23:56 +01:00
Gunnar Morling
c9c84afc9d DBZ-1099 Don't enque further records after request to stop 2019-01-25 10:19:19 +01:00
Gunnar Morling
95a13b5dd7 DBZ-1062 Fix MySQL trustStore password setting 2019-01-24 17:38:42 +01:00
Moira Tagle
01c779fb62 DBZ-175 Setting completed GTIDs in the final binlog reader context in setupUnifiedReader() 2019-01-24 10:43:44 +01:00
Moira Tagle
0e10c4af48 DBZ-175 Modifying BinlogReaderMetrics to take in the BinlogReader name 2019-01-24 10:43:44 +01:00
Gunnar Morling
884f2f40ea DBZ-175 Adding toString() for readers 2019-01-24 10:43:44 +01:00
Gunnar Morling
206cd06c24 DBZ-175 Using passed filters instead of rebuilding them in MySqlSchema 2019-01-24 10:43:44 +01:00
Gunnar Morling
ed4787bc8a DBZ-175 Misc. clean-up; 
* Removing unused code (constructor, method and return value)
* Making ParallelSnapshotReader logger static
* Making RecordMakers#restartOffset final; avoiding unchecked cast
 2019-01-24 10:43:44 +01:00
Gunnar Morling
965b789dab DBZ-175 Misc. adjustments; 
* Dedicated functional interface for halting predicates
* Using semantic temporal types
* Typo fixes
* Exposing SnapshotNewTables through connector config
* Static loggers
* Adding Moira to COPYRIGHT.txt
 2019-01-24 10:43:44 +01:00
Moira Tagle
a6d791cd8c DBZ-175 Initial implementation for support to whitelist/blacklist changes; 
* The filter config is stored in offsets now, allowing to detect changes
to the config after a connector restart
* In that case, roughly the following steps are done:
- set up a binlog reader for the "old" tables
- set up a snapshot reader for the "new" tables
- if the snapshot is done, set up a binlog reader for the new tables
- if both binlog readers are in proximity of the logs head, stop them
and set up a single binlog reader for all tables
* That behavior is disabled by default for the purposes of testing and
can be enabled via new connector option "snapshot.new.tables"
* To facilitate connector restarts while whitelist/blacklist changes are
processed, separate restart offsets are added to the offsets
 2019-01-24 10:43:44 +01:00
Jiri Pechanec
a638d5599b DBZ-1064 Use geometry instead of point for empty 2019-01-16 13:01:36 +01:00
Jiri Pechanec
ceb6c62721 DBZ-1064 Remove lazy fallback values 2019-01-16 13:01:36 +01:00
Jiri Pechanec
536e517c08 DBZ-1064 Geometry default value changes; fixes for default suppliers 2019-01-16 13:01:36 +01:00
Jiri Pechanec
bff7eea734 DBZ-1064 Use convert value template 2019-01-16 13:01:36 +01:00
Jiri Pechanec
920e0cbd3f DBZ-1064 Set default value only for non-nullable fields 2019-01-16 13:01:36 +01:00
Amit Sela
b0b7d942a3 DBZ-1073 Adding scale as schema parameter, if present 2019-01-11 15:40:30 +01:00
Grzegorz Kołakowski
91c72738c0 DBZ-1067 Add column blacklist field to RelationalDatabaseConnectorConfig 2019-01-09 18:13:26 +01:00
Gagan Agrawal
aff45c7dc6 DBZ-1063 Fixed schema parsing bug for handling unsigned max default value 2019-01-09 17:50:18 +01:00
Jiri Pechanec
03570a1a3a DBZ-1059 Support qualified table name in rename 2019-01-04 12:57:55 +01:00
Jenkins user
5275f73424 [maven-release-plugin] prepare for next development iteration 2018-12-19 13:06:12 +00:00
Jenkins user
b6569c18ae [maven-release-plugin] prepare release v0.9.0.Beta2 2018-12-19 13:06:12 +00:00
Jiri Pechanec
9fcc25e4c5 DBZ-688 Uncommitted TX with temp tables are not stored in binlog in MySQL 8 2018-12-18 15:42:15 +01:00
LiuHanlin
c58031582d DBZ-1033 Mysql binlog reader lost data if restart task when last binlog event is QUERY event. 
https://issues.jboss.org/browse/DBZ-1033
Reset restartEventsToSkip each time updating the restartBinlogPosition.
 2018-12-17 13:42:01 +01:00
Gunnar Morling
1258beae55 DBZ-688 Moving differences related solely to geo-spatial test into the test class itself 2018-12-14 10:39:05 +01:00
Jiri Pechanec
c8f09f110b DBZ-688 Centralize MySQL differences in a single file 2018-12-14 10:39:05 +01:00
Jiri Pechanec
3c1da7d5d4 DBZ-688 Robust identification of geom collection type 2018-12-14 10:39:05 +01:00
Jiri Pechanec
12083452a5 DBZ-688 CURRNT_TIMESTAMP handled differently by MySQL 8 2018-12-14 10:39:05 +01:00
Jiri Pechanec
994f1b6e03 DBZ-688 Update geometry handling 2018-12-14 10:39:05 +01:00
Jiri Pechanec
8b83df82b5 DBZ-688 Ignore differences in generated metadata 2018-12-14 10:39:05 +01:00
Jiri Pechanec
974731b0ae DBZ-688 Support for test execution on MySQL 8 server 2018-12-14 10:39:05 +01:00
Jiri Pechanec
edec1c3090 DBZ-978 Added last event and captured tables metrics 2018-12-13 20:52:11 +01:00
Grzegorz Kołakowski
2a1934eeb1 DBZ-966 Make snapshot.delay.ms global property 2018-12-13 14:29:54 +01:00
Gunnar Morling
0075bbb17e DBZ-1028, DBZ-992 Expanding test 2018-12-12 23:05:29 +01:00
Jiri Pechanec
946d5c113d DBZ-1028 Literal "CSV" is not a special token 2018-12-12 23:05:07 +01:00
Jiri Pechanec
f3a87a039f DBZ-1008 Process first GTID after failover 2018-12-12 16:52:53 +01:00
Jiri Pechanec
edb116677d DBZ-960 Handle TX closing safely 2018-12-11 14:15:53 +01:00
Jiri Pechanec
2b2ff0754d DBZ-960 Run SnapshotReader in a single transaction 2018-12-11 14:15:53 +01:00
Gunnar Morling
68b6bc2efe DBZ-977 Adding log statement 2018-12-11 14:12:23 +01:00
Jiri Pechanec
6a88309f0e DBZ-977 ALTER TABLE filters works even for stored DDL 2018-12-11 14:12:23 +01:00
Jiri Pechanec
c5ecff4be6 DBZ-1005 Detect GTID by database setting not GTID set 2018-11-30 11:58:07 +01:00
Jiri Pechanec
415830003a DBZ-1016 Fix expected records count 2018-11-30 11:18:47 +01:00
Jiri Pechanec
2c3d2ee083 DBZ-1010 Handle error deserialization also in deserializers 2018-11-30 11:09:30 +01:00
Jiri Pechanec
9c81e54b9d DBZ-1006 Guard against NPE when start() fails 2018-11-23 16:13:50 +01:00
Gunnar Morling
62fe7d1ee7 DBZ-604 Making sure connector cleanly shuts down also if records are produced after polling loop has stopped 2018-11-23 14:36:05 +01:00
Gunnar Morling
2fd16a6bc0 DBZ-923 Misc. clean-up; 
* Exposing new option via MySqlConnectorConfig
* Formatting
* Using GtidSet type in signatures
* Adding new option to configDef() and ALL_FIELDS
 2018-11-22 10:54:10 +01:00
Eero Koplimets
972c08fcda fix checkstyle errors 2018-11-22 09:50:46 +01:00
Eero Koplimets
3e1eaef318 fix GTID_NEW_CHANNEL_POSITION description 2018-11-22 09:50:46 +01:00
Eero Koplimets
268101acd5 Change config from boolean to enum 2018-11-22 09:50:46 +01:00
Eero Koplimets
8844442459 DBZ-923 config to use earliest available gtid set from mysql server when connecting 2018-11-22 09:50:46 +01:00
Jenkins user
db1d3a7fb8 [maven-release-plugin] prepare for next development iteration 2018-11-20 16:15:14 +00:00
Jenkins user
f83db82cea [maven-release-plugin] prepare release v0.9.0.Beta1 2018-11-20 16:15:14 +00:00
Jiri Pechanec
b7e7bfa218 DBZ-990 Engine names can be quoted 2018-11-19 14:30:21 +01:00
Jiri Pechanec
530e2190a8 DBZ-757 Use antlr parser by default 2018-11-16 14:17:47 +01:00
Gunnar Morling
6bd4024618 DBZ-776 Misc. improvements; 
* Renaming getTimeSinceLastEvent() to getMilliSecondsSinceLastEvent()
* Further unifying metrics implementation across connectors
* Emitting event in EventDispatcher also if event is filtered out
* Typo fixes
 2018-11-16 08:52:07 +01:00
Gunnar Morling
818d5e4696 DBZ-776 Misc. clean-up 2018-11-16 08:52:07 +01:00
Jiri Pechanec
4510e2bbe6 DBZ-776 Cleanup after review 2018-11-16 08:52:07 +01:00
Gunnar Morling
360714a5a1 DBZ-776 Formatting and minor clean-up 2018-11-16 08:52:07 +01:00
Jiri Pechanec
b44c9fd04f DBZ-776 A first set of metrics implemented in new framework 2018-11-16 08:52:07 +01:00
Jiri Pechanec
4cdc3e669b DBZ-776 Extract metrics to a common code 2018-11-16 08:52:07 +01:00
Gunnar Morling
9994172860 DBZ-953 Pulling decimal mode handling up to RelationalDatabaseConnectorConfig; 
Also using it in MySQL connector.
 2018-11-08 09:23:28 +01:00
Jiri Pechanec
a5ff561deb DBZ-959 ADD PARTITION needs parenthesis 2018-11-02 07:14:42 +01:00
s0r01ed
fdab3386d4 DBZ-780 | Adding RENAME column semantics to rename a column 2018-10-24 20:20:04 +02:00
Deepak Barr
bc0712e240 DBZ-935 Error while parsing json column type for MySQL 2018-10-11 11:40:18 +02:00
Jenkins user
d952f5dfb0 [maven-release-plugin] prepare for next development iteration 2018-10-04 11:59:14 +00:00
Jenkins user
ff9b70278b [maven-release-plugin] prepare release v0.9.0.Alpha2 2018-10-04 11:59:13 +00:00
Deepak Barr
4aaed1fd2b DBZ-927 Gracefully handling invalid default values for timestamp fields 2018-10-04 10:30:56 +02:00
jchipmunk
3ec0b3be81 DBZ-918 Moving connector identifier from SourceInfo to Module class 2018-10-04 09:00:13 +02:00
jchipmunk
05ba99bb0a DBZ-918 Adding Debezium connector field to source info 
This will allow consumers to recognize the Debezium connector used for creating a given message, helping them to adjust their behavior for a variety of connectors.
 2018-10-04 09:00:13 +02:00
Jiri Pechanec
3b51818864 DBZ-910 Verify also anonymous CONSTRAINT CHECK 2018-09-18 15:51:46 +02:00
Jiri Pechanec
3c5a67856b DBZ-910 Improve support for CHECK constraint 2018-09-18 15:51:46 +02:00
Jiri Pechanec
6fe48b7764 DBZ-769 Try to release table locks during exception 2018-09-18 10:49:48 +02:00
jchipmunk
b0feaf496a DBZ-633 Adding "field.blacklist" property for MongoDB connector 
The "field.blacklist" configuration property is an optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are of the form "databaseName.collectionName.fieldName.nestedFieldName", where "databaseName" and "collectionName" may contain the wildcard (*) which matches any characters.

Although the "field.blacklist" configuration property allows you to remove fields from the event values, the "_id" field is always included in the event’s key.
 2018-09-14 12:54:11 +02:00
Jiri Pechanec
e6b28eeea4 DBZ-903 Support function named DATABASE() 2018-09-13 16:17:50 +02:00
Jiri Pechanec
8dd8327826 DBZ-901 Handle default date/time values without micros 2018-09-13 15:47:00 +02:00
Gunnar Morling
e1bbbd7992 DBZ-865 DBZ-878 Replacing invalid topic name characters with "_"; 2018-09-05 12:56:53 +02:00
Cliff Wheadon
685f72f988 DBZ-880 MySQLConnectorTask no longer always reports finding the binlog 2018-08-28 08:56:52 +02:00
Jiri Pechanec
ce0b3cc940 DBZ-872 Use converters in parser tests 2018-08-24 08:12:54 +02:00
Jiri Pechanec
2cbc35dac7 DBZ-877 Process newly added columns 2018-08-24 08:12:54 +02:00
Gunnar Morling
985999b4af DBZ-872 Making MySqlAntlrDdlParser#tableFilter final 2018-08-24 08:12:54 +02:00
Jiri Pechanec
63ee65bb7d DBZ-872 Honor filters in ALTER TABLE parsing 2018-08-24 08:12:54 +02:00
Jiri Pechanec
29af79057b DBZ-859 Test for MySQL heartbeat 2018-08-23 11:44:53 +02:00
Jiri Pechanec
985e03e7b5 DBZ-859 Force heartbeat after snapshot completion 2018-08-23 11:44:53 +02:00
Rao
49214491a7 DBZ-763 Correcting the JDBC driver class 2018-08-23 11:31:37 +02:00
Gunnar Morling
414ebe5f13 DBZ-870 Adding test 2018-08-23 11:19:27 +02:00
Jiri Pechanec
99ab131910 DBZ-870 Accept zero as valid date/time value 2018-08-23 11:19:27 +02:00
Jiri Pechanec
26b82c88bd DBZ-860 DBZ-869 Enable default value tests for antlr parser 2018-08-23 11:19:27 +02:00
Jiri Pechanec
e9f9e4eb8f DBZ-860 Fix for primary keys with default value 2018-08-23 11:19:27 +02:00
Jiri Pechanec
7e0639f733 DBZ-864 Add support for CHARSET database option 2018-08-22 13:31:34 +02:00
Gunnar Morling
207f1896ab DBZ-763 Cleaning up tests 2018-08-22 11:54:55 +02:00
Rao
d13ea932f7 DBZ-763 Upgrading Mysql driver to version 8.0.12 2018-08-22 11:54:55 +02:00
Gunnar Morling
3862083756 DBZ-860 Expanding test 2018-08-20 19:01:55 +02:00
Jiri Pechanec
c2f014750e DBZ-860 Handle implicitly NOT NULL PKs 2018-08-20 19:01:32 +02:00
PengLv
3d978d717b fix gitd bug 
method isGtidModeEnabled when to check gtid mode ,it'll always be true
 2018-08-14 09:46:16 +02:00
artiship
bb4ecc79e0 DBZ-849 Refact read mysql system variables methods 2018-08-14 09:43:49 +02:00
artiship
cdb6d7301c DBZ-855 Make BinlogReader.informAboutUnknownTableIfRequired log with tableId 2018-08-14 08:41:30 +02:00
Jenkins user
e00dad127f [maven-release-plugin] prepare for next development iteration 2018-07-26 08:00:12 +00:00
Jenkins user
16bfd5c700 [maven-release-plugin] prepare release v0.9.0.Alpha1 2018-07-26 08:00:12 +00:00
Gunnar Morling
3061c1dc0b DBZ-815 Passing offset to Heartbeat#heartbeat(); 
That's simpler to grasp than the approach of passing a supplier lambda to the constructor.
Also it allows to pass on the offset via local variables instead of instance fields in some cases.
 2018-07-25 14:49:52 +02:00
Jiri Pechanec
0e35439019 DBZ-40 Snapshotting and connector restart 2018-07-18 14:39:11 +02:00
Gunnar Morling
20db9299c5 DBZ-627 Unifying TopicSelector implementations 2018-07-18 12:36:07 +02:00
Gunnar Morling
1523f230ca DBZ-793 Using TableFilter consistently; 
Only the MySQL connector's Filters class still uses Predicate for the time being, so to avoid to much merging trouble with DBZ-175.
 2018-07-11 13:16:44 +02:00
Jenkins user
f9b8d830a8 [maven-release-plugin] prepare for next development iteration 2018-07-11 07:36:30 +00:00
Jenkins user
290ded678f [maven-release-plugin] prepare release v0.8.0.Final 2018-07-11 07:36:29 +00:00
Peter Goransson
448330fd07 DBZ-789 Expose more granular snapshot metrics via JMX 2018-07-09 12:30:41 +02:00
Jenkins user
033db6659d [maven-release-plugin] prepare for next development iteration 2018-07-04 07:07:44 +00:00
Jenkins user
696f35f2c0 [maven-release-plugin] prepare release v0.8.0.CR1 2018-07-04 07:07:44 +00:00
Jiri Pechanec
d4c9d24b22 DBZ-779 Detect mongo replicator init completion 2018-07-03 12:24:06 +02:00
Gunnar Morling
9d52614b79 DBZ-771 Adding test for changing column type and default value 2018-07-03 11:55:20 +02:00
Gunnar Morling
99048365c9 DBZ-771 Unsetting default value when changing a column's type 2018-07-03 11:55:20 +02:00
Gunnar Morling
dcd6b01134 DBZ-644 Simplifying RelationalDatabaseSchema constructor; 
Also retrieving logical name via CommonConnectorConfig#getLogicalName() in more places
 2018-07-03 07:03:59 +02:00
Gunnar Morling
068aa85bd6 DBZ-644 Misc. clean-up and tests; 
* Using parameter keys under "__debezium" namespace
* More expressive names
* Adding tests
 2018-07-03 07:03:59 +02:00
Gunnar Morling
8744488c8a DBZ-644 Unifying retrieval of ColumnMappers in RelationalDatabaseSchema 2018-07-03 07:03:59 +02:00
orr.ganani
747184c572 DBZ-644 add JDBC data type as additional metadata to DBZ events, if configuration is opted in 2018-07-03 07:03:59 +02:00
Gunnar Morling
c4c8cbc3ab DBZ-773 Moving management of Tables to RelationalDatabaseSchema 2018-06-29 09:09:10 +02:00
Gunnar Morling
bbfbdf6fab DBZ-773 Moving management of table schemas to RelationalDatabaseSchema 2018-06-29 09:09:10 +02:00
Gunnar Morling
90a453b0ee DBZ-768 Checking boolean flag just once 2018-06-28 10:14:01 +02:00
Jiri Pechanec
b4b1f6976b DBZ-768 Fix formatting. Behaviour not dependent on a timezone 2018-06-28 07:24:09 +02:00
Jiri Pechanec
741debe18e DBZ-768 Default values with variable second fractions 2018-06-28 06:17:41 +02:00
Gunnar Morling
a3f42610ff DBZ-751 Adding tests for NUMERIC columns 2018-06-27 14:48:59 +02:00
Gunnar Morling
0600ccf213 DBZ-751 Incorporating review remarks; 
* Expanding test
* Extracting constant for parameter key in tests
 2018-06-27 14:48:59 +02:00
Gunnar Morling
a292b05a96 DBZ-751 Propagating DECIMAL column precision to Avro schemas 2018-06-27 14:48:59 +02:00
Jiri Pechanec
d61db8930d DBZ-770 Typename comparison should not be case dependent 2018-06-27 11:37:58 +02:00
Jiri Pechanec
ca6b1c9168 DBZ-743 Empty password for PostgreSQL and MySQL 2018-06-27 11:18:19 +02:00
Gunnar Morling
9d05fc2dc5 DBZ-767 Typo fix 2018-06-27 10:34:00 +02:00
Gunnar Morling
20f622dc40 DBZ-767 Adjusting test to more closely match original report 2018-06-27 10:33:41 +02:00
Jiri Pechanec
84f8bfb070 DBZ-767 Column should not be renamed when name is not changed 2018-06-27 09:14:59 +02:00
Jiri Pechanec
c14693c4ec DBZ-765 Reduce the amount of test prints 2018-06-25 10:45:46 +02:00
Gunnar Morling
bf7a5018ca Setting POM version back to 0.8.0-SNAPSHOT 2018-06-22 12:21:03 +02:00
Jenkins user
db42e4657a [maven-release-plugin] prepare for next development iteration 2018-06-21 14:07:45 +00:00
Jenkins user
c4b8ecaa99 [maven-release-plugin] prepare release v0.8.0.Beta1 2018-06-21 14:07:45 +00:00
Gunnar Morling
f5629d9f0b DBZ-578 Interpret DATETIME columns using UTC; 
This partly reverts the work previously done for DBZ-578,
where DATETIME columns (logical date + time values without TZ)
were shifted using the database's (or connection's) time zone.
Instead, the DATETIME values are interpreted using UTC again
with this change.
 2018-06-20 15:58:19 +02:00
Gunnar Morling
42e313284a DBZ-20 Adjustment after rebase 2018-06-20 13:05:37 +02:00
Gunnar Morling
b83f1493a2 DBZ-20 Avoiding Optional as method parameter 2018-06-20 13:05:37 +02:00
Gunnar Morling
dce1aa29f1 DBZ-20 Oracle precision supports negative numbers 2018-06-20 13:05:37 +02:00
Gunnar Morling
603f02e70d DBZ-20 Excluding generated sources from CheckStyle 2018-06-20 13:05:37 +02:00
Gunnar Morling
d7e196a18e DBZ-20 Initial import of Oracle connector based on XStream 2018-06-20 13:05:37 +02:00
Gunnar Morling
5b1807b286 DBZ-734 Avoiding repetition of option name and default value 2018-06-18 09:51:01 +02:00
Jiri Pechanec
ec03b431c9 DBZ-734 Maven profile for testing old/new DDL parser 2018-06-18 06:02:29 +02:00
Gunnar Morling
9eb4b90ec9 DBZ-191 Adding record validations 2018-06-15 20:23:17 +02:00
Gunnar Morling
84dd36df46 DBZ-252 Misc typo fixes and clean-up 2018-06-15 16:43:30 +02:00
Gunnar Morling
b56d6efdf1 DBZ-252 Making DataTypeResolver immutable 2018-06-15 16:43:29 +02:00
Gunnar Morling
54ca30624d DBZ-252 Misc. improvements; 
* Dedicated getter for DDL mode
* Using Objects#equal()
* Typo fixes
 2018-06-15 16:43:29 +02:00
Gunnar Morling
6a6a8c9f99 DBZ-727 Initializing defaults for NUMERIC in Antlr parser 2018-06-15 16:43:29 +02:00
Gunnar Morling
759940b879 DBZ-252 Using existing parser implementation by default; 
* Avoiding reflection for parser instantiation
* Making legacy parser instantiatable again
 2018-06-15 16:43:25 +02:00
Jiri Pechanec
9580c8c290 DBZ-252 Rebase to master 2018-06-15 11:42:24 +02:00
rkuchar
5022933581 DBZ-252 add javadoc 2018-06-15 11:42:24 +02:00
rkuchar
c2e13b3b5f DBZ-252 minor refactor 2018-06-15 11:42:24 +02:00
rkuchar
98384822eb DBZ-252 refactor - move mysql antlr ddl parsser implementation into mysql-connector module 2018-06-15 11:42:24 +02:00
rkuchar
b8191963f7 DBZ-252 resolve todos 2018-06-15 11:42:24 +02:00
rkuchar
6812b4f7f0 DBZ-252 remove unused import 2018-06-15 11:42:24 +02:00
rkuchar
5c9e718436 DBZ-252 uncomment getting ddl parser instance from configuration + improve/refactor dataType resolver for ANTLR 2018-06-15 11:42:24 +02:00
rkuchar
99eac9a8e2 DBZ-252 some test fixes + separate ddl test for antlr parser 2018-06-15 11:42:24 +02:00
rkuchar
a4105adaaa DBZ-252 refactor skipping nodes in tree walk 2018-06-15 11:42:24 +02:00
rkuchar
1ae7859acd DBZ-252 parse create view 2018-06-15 11:42:23 +02:00
rkuchar
07212043b5 DBZ-252 test fixes + grammar fixes + some bug repairs in antlr parser 2018-06-15 11:42:23 +02:00
rkuchar
b4480c55b7 DBZ-252 parse database queries (create, alter, delete), parse name tokens without quotes, add new Mysql connector configuration field for choosing ddl parsing mode 2018-06-15 11:42:23 +02:00
rkuchar
32d181480d DBZ-252 fix troubles with generalized system variables + add new data types that was missing in grammar 2018-06-15 11:42:23 +02:00
rkuchar
5851d1f07d DBZ-252 fix initialization of system variables object + add default scope for variales in case that scope will no be set 2018-06-15 11:42:23 +02:00
rkuchar
9311c82bc5 DBZ-252 reformat code to not have anything after closing "}" 2018-06-15 11:42:23 +02:00
rkuchar
782ab75160 DBZ-252 move listener support from base to only legacy implementation, refactor system variables for possible use with other DBMSs + introduce data type resolver for antlr mysql parser 2018-06-15 11:42:23 +02:00
rkuchar
6a21702ac9 DBZ-252 finished alter table query listeners + new listener for rename table and create index 2018-06-15 11:42:23 +02:00
rkuchar
909668e14a DBZ-252 Mysql alter table by add column/s parser listener 2018-06-15 11:42:23 +02:00
rkuchar
1df0b2033c DBZ-252 Create new debezium-ddl-parser module 2018-06-15 11:42:23 +02:00
rkuchar
36e2f33bf6 DBZ-252 - extract creation of DdlParser interface 2018-06-15 11:42:23 +02:00
rkuchar
30cb9c3059 DBZ-252 - remove UNDERSCORE fragment from lexer 2018-06-15 11:42:23 +02:00
rkuchar
4a70d9bd59 DBZ-252 - case insensitive lexer 2018-06-15 11:42:23 +02:00
rkuchar
fb53cde565 DBZ-252 = add ANTLR gramatic for MySQLm setup build for it and exclude generated files from checkstyle 2018-06-15 11:42:23 +02:00
Jiri Pechanec
f9c84030b3 DBZ-727 Initializing defaults for NUMERIC 2018-06-14 16:39:48 +02:00
Jiri Pechanec
eadf65cd27 DBZ-578 Build failing on Jenkins CI 2018-06-13 14:43:29 +02:00
Jiri Pechanec
29f8891b4f DBZ-578 Process timestamps without timezones as in server timezone 2018-06-11 15:27:08 +02:00
luobo
3f7186b3e5 DBZ-191 Represent all-zero time value with null if column allows null 2018-06-11 10:01:22 +02:00
Peter Goransson
3bf3e2e989 DBZ-601 Exposing new config 2018-06-07 10:42:20 +02:00
Gunnar Morling
c5b907873b DBZ-601 Minor clean-up 2018-06-06 08:40:05 +02:00
Peter Goransson
f387f72474 DBZ-601 Snapshot delay implementation 2018-06-06 08:33:10 +02:00
luobo
5b33d81f00 DBZ-191 support default value of column in mysql 2018-06-04 10:43:53 +02:00
Gunnar Morling
61f576bb90 DBZ-706 Misc. clean-up 2018-06-01 11:23:30 +02:00
Stephen Powis
40d872f48c DBZ-706 General cleanup 2018-06-01 11:23:30 +02:00
Stephen Powis
867202e0ca DBZ-706 Slight refactoring / shuffling of code to validate source query field in snapshot tests 2018-06-01 11:23:12 +02:00
Stephen Powis
cc33fed16a DBZ-706 Adding option for showing source query in sourceInfo element 2018-06-01 11:23:12 +02:00
Gunnar Morling
8d7806e71a DBZ-693 Expanding test; updating connector option description 2018-05-28 12:35:47 +02:00
Jiri Pechanec
45bc7b1536 DBZ-693 On connect statements supported for PostgreSQL 2018-05-28 12:35:47 +02:00
Jiri Pechanec
f74a9fe059 DBZ-693 Initial SQL statements 2018-05-28 12:35:47 +02:00
Jiri Pechanec
fc107972f2 DBZ-598 Metrics expose the list of captured tables 2018-05-28 09:40:33 +02:00
Rao
15a90548fb DBZ-666 Supporting ordered snapshot using tables.whitelist config 2018-05-17 06:23:31 +02:00
vucina.ivan
cd78b873fc DBZ-670 Adding connect.keep.alive.interval.ms configuration option 2018-03-28 09:19:13 +02:00
Rao
1e1946d77e DBZ-639 Removing restriction for none snapshot locking mode so that it can be used for all snapshot modes 2018-03-22 10:36:01 +01:00
Jiri Pechanec
a84b7ca682 DBZ-667 Test for scientific value sent to binlog 2018-03-20 14:42:55 +01:00
Jiri Pechanec
2b5370de1a DBZ-667 Sceintific numbers parsing 2018-03-20 14:42:55 +01:00
Jenkins user
f4e151b23a [maven-release-plugin] prepare for next development iteration 2018-03-20 08:14:19 +00:00
Jenkins user
93b3252332 [maven-release-plugin] prepare release v0.7.5 2018-03-20 08:14:19 +00:00
Jiri Pechanec
40a527b6ea DBZ-592 Make snapshot/init-sync false instead of null 2018-03-19 13:19:34 +01:00
Maciej Bryński
b7f13213b2 DBZ-646 Enhancing DDL parser to support compression parameter used with 
CREATE TABLE on TokuDB
 2018-03-18 18:29:41 +01:00
Peter Goransson
21cdf77634 DBZ-640 Destroy / de-initialization for Readers 2018-03-16 10:15:08 +01:00
Jiri Pechanec
469199350f DBZ-660 Fix for CONSTRAINT UNIQUE syntax 2018-03-16 09:44:47 +01:00
Jenkins user
daf27207be [maven-release-plugin] prepare for next development iteration 2018-03-07 08:31:07 +00:00
Jenkins user
9c73774928 [maven-release-plugin] prepare release v0.7.4 2018-03-07 08:31:07 +00:00
Jiri Pechanec
a51322da0c DBZ-611 Removed superfluous conversion 2018-03-06 16:55:24 +01:00
Jiri Pechanec
937c8b82f8 DBZ-611 Removed misleading comment 2018-03-06 16:55:24 +01:00
Jiri Pechanec
424aefbbb4 DBZ-611 Decimal-> String supported for MySQL 2018-03-06 16:55:24 +01:00
Gunnar Morling
f055020613 DBZ-602 Typo fixes; adding a few test assertions 2018-03-06 15:38:59 +01:00
stephen powis
c11875994a DBZ-602 Adding new locking mode for bypassing table locks 2018-03-06 15:08:55 +01:00
Gunnar Morling
9b88e035f5 DBZ-641 Formatting 2018-03-02 09:05:07 +01:00
Satyajit Vegesna
e79e55d07f DBZ-641 Enhancing parser to support CREATE TABLE with PARTITION ... ENGINE=InnoDB 2018-03-02 09:00:31 +01:00
Gunnar Morling
d89f4e91c6 DBZ-632 Removing unused method parameter 2018-02-23 21:54:41 +01:00
Gunnar Morling
c35c1827ff DBZ-630 Refactoring around MySQL task context; 
* Making relationship from MySqlTaskContext to MySqlJdbcContext has-a i
 2018-02-21 12:04:41 +01:00
Gunnar Morling
eef1ad7c2a DBZ-630 Refactoring around MongoDB task context; 
* Renaming ConnectorTaskContext to CdcSourceTaskContext
* Renaming ReplicationContext to MongoDbTaskContext
* Making relationship from MongoDbTaskContext to ConnectionContext has-a instead of is-a
 2018-02-21 12:04:41 +01:00
Gunnar Morling
bb12e521f8 DBZ-630 Pulling up getClock() to ConnectorTaskContext 2018-02-21 12:04:41 +01:00
Gunnar Morling
8264c20cf6 DBZ-630 Unifying common start-up logic across connectors 2018-02-21 12:04:41 +01:00
Gunnar Morling
7fe45db831 DBZ-626 Renaming AvroValidator to SchemaNameAdjuster; 
Making apparent that this functionality doesn't solely validate but returns a new name actually
 2018-02-20 09:26:47 +01:00
Gunnar Morling
cddd5fcc8a DBZ-626 Passing schema validator as AvroValidator instance instead of Function 2018-02-20 09:26:47 +01:00
Jiri Pechanec
a08a00d46e DBZ-615 Init unset Decimal props to default values 2018-02-20 08:54:28 +01:00
Jiri Pechanec
a06465ae34 DBZ-629 Fixed intermittent test issues 2018-02-20 08:01:51 +01:00
Gunnar Morling
e895fff2ac DBZ-628 Using common properties for MySQL connector 2018-02-20 06:34:42 +01:00
Gunnar Morling
ab11f48843 DBZ-620 Creating envelope schemas only once per table type 2018-02-19 12:18:37 +01:00
Jenkins user
6d0cd88e12 [maven-release-plugin] prepare for next development iteration 2018-02-15 04:15:34 +00:00
Jenkins user
7d1e1a989e [maven-release-plugin] prepare release v0.7.3 2018-02-15 04:15:34 +00:00
Jiri Pechanec
2e4be4c5ee DBZ-278 Create topic only when needed 2018-02-13 16:48:42 +01:00
Jiri Pechanec
4541fc53c9 DBZ-278 Explicit history storage initialization 2018-02-13 16:48:42 +01:00
Jiri Pechanec
ecdce6529d DBZ-220 Null Heartbeat object. Messages contains key 2018-02-13 12:26:48 +01:00
Jiri Pechanec
682cabb84a DBZ-220 Generate heartbeat events in binlog thread 2018-02-13 12:24:51 +01:00
Jiri Pechanec
3c1ea8206b DBZ-220 Refactor heartbeat to a separate class 2018-02-13 12:24:51 +01:00
Jiri Pechanec
4a22e17b71 DBZ-220 Disable heartbeat by default 2018-02-13 12:22:18 +01:00
Jiri Pechanec
ad682e109d DBZ-220 Constatnts configurable, heartbeat independent of binlog 2018-02-13 12:22:18 +01:00
Jiri Pechanec
e6936616d9 DBZ-220 Send periodically messages to a heartbeat topic 2018-02-13 12:19:30 +01:00
Gunnar Morling
3e29591846 DBZ-582 Supporting TOMBSTONES_ON_DELETE option with MySQL connector 2018-02-13 10:19:18 +01:00
Gunnar Morling
e8fa7b887e DBZ-588 Typo fix and additional method comment 2018-02-09 14:16:53 +01:00
Jiri Pechanec
2710094d11 DBZ-588 Defensively protecting CUD events themeselves 2018-02-09 11:03:16 +01:00
Jiri Pechanec
e4c9cf7191 DBZ-588 Changed message wording 2018-02-09 11:03:16 +01:00
Jiri Pechanec
16f4726a31 DBZ-588 Fail on inconsistent schema 2018-02-09 11:03:16 +01:00
Gunnar Morling
712824e43f DBZ-593 Passing version explicitly from SourceInfos to base class instead of relying specific packaging structure 2018-02-02 13:16:43 +01:00
Jiri Pechanec
7ebee94169 DBZ-593 Add Debezium version to source in envelope 2018-02-02 13:16:43 +01:00
Gunnar Morling
3a281d1185 DBZ-595 Making sure resources are cleaned up when snapshotting fails; 
* shutting down the snapshotting thread and the DB history producer client
  if the connector is stopped while trying to write to the history topic
* reducing the time that KafkaProducer#send() will block if Kafka isn't
  available; this will release the producer thread quicker in case the
  connector is stopped during snapshotting
* not returning from finally block (!) in case the TX is rolled back; This
  prevented the failed state to be set by the outer catch clause in execute()
 2018-02-02 09:06:32 +01:00
Gunnar Morling
8821d89dfe DBZ-587 Making sure snapshot reader thread is removed 2018-02-01 10:04:20 +01:00
Jiri Pechanec
92740a3626 DBZ-587 Fix thread leak, thread names 2018-02-01 10:04:20 +01:00
Jiri Pechanec
9b592204ac DBZ-587 Centralize and unify thread management 2018-02-01 10:04:20 +01:00
Gunnar Morling
018a879d40 DBZ-583 Extracting a builder from ChainedReader, avoiding mutable state within its implementation 2018-01-30 08:59:41 +01:00
Jiri Pechanec
93cfb4f4cb DBZ-572 Transaction is completed also by XID event 2018-01-29 15:40:41 +01:00
Jenkins user
04624341f5 [maven-release-plugin] prepare for next development iteration 2018-01-25 09:39:44 +00:00
Jenkins user
898f6884e1 [maven-release-plugin] prepare release v0.7.2 2018-01-25 09:39:44 +00:00
Robert Coup
f9d90a482f DBZ-507 Expanding support for geometry types; 
* Adding support for PostGIS geometry types
* Adding support for GEOMETRY, POLYGON and more in MySQL
* For all newly supported types, changes are represented using two new schema types Geometry and Geography, containing the WKB (binary geo data) ans srid (coord system identifier)
* The existing Point type also contains the new (optional) srid field
 2018-01-24 16:51:49 +01:00
Jiri Pechanec
01577b40c3 DBZ-541 Only whitelisted tables are recorded in history 2018-01-24 14:43:57 +01:00
Peter Goransson
7d78357087 DBZ-443 Adding db-history recovery mode: schema_only_recovery 2018-01-23 09:17:39 +01:00
Gunnar Morling
7a7d43c237 DBZ-516 Returning control also from BlockingReader 2018-01-18 14:13:58 +01:00
Gunnar Morling
0c4190c493 DBZ-516 Using Duration instead of long in a few more places 2018-01-18 14:13:58 +01:00
Jiri Pechanec
24bdcaf059 DBZ-516 Return control to Connect periodically 2018-01-18 14:13:58 +01:00
Gunnar Morling
1d967a6d47 DBZ-392 Propagating value of tableIdCaseInsensitive 2018-01-17 11:21:02 +01:00
Gunnar Morling
4d1d016acb DBZ-392 Making table map and set dedicated classes just with the needed API 2018-01-17 11:21:02 +01:00
Jiri Pechanec
9f0f0fe5d4 DBZ-392 Support for MySQL running on lowercase filesystems 2018-01-17 11:21:02 +01:00
Jiri Pechanec
446b102ad9 DBZ-534 Using correct datattype in list lookup 2018-01-12 12:30:44 +01:00
Jiri Pechanec
0b269a6e41 DBZ-538 Improve invalid DDL statement reporting 2018-01-12 12:26:30 +01:00
Jiri Pechanec
374999ee45 DBZ-411 ROLLBACK TO savepoint is parseable 2018-01-11 09:19:11 +01:00
Jiri Pechanec
3d7cc6a7e3 DBZ-530 Multiple PARTITION defintions supported 2018-01-08 11:58:09 +01:00
Jiri Pechanec
42991d260f DBZ-524 RESTRICT in DROP COLUMN 2018-01-02 09:26:07 +01:00
Peter Goransson
5f3902d240 DBZ-522 Adding null check when encounting null TIME fields during snapshot 2018-01-02 07:00:03 +01:00
Jenkins user
6bb34b42f9 [maven-release-plugin] prepare for next development iteration 2017-12-20 07:15:12 +00:00
Jenkins user
16dcd4c980 [maven-release-plugin] prepare release v0.7.1 2017-12-20 07:15:12 +00:00
Jenkins user
5e09932cb9 [maven-release-plugin] prepare for next development iteration 2017-12-15 05:10:23 +00:00
Jenkins user
6c1d61e03b [maven-release-plugin] prepare release v0.7.0 2017-12-15 05:10:23 +00:00
Jiri Pechanec
196f6b3571 DBZ-406 Fixing test, adding warning for disabled rollback 2017-12-13 14:12:27 +01:00
Gunnar Morling
06e72308d9 DBZ-406 Misc improvements; 
* Making isReplayingEventsBeyondBufferCapacity() a "query method" only
* Using constant for buffer default size
* Misc. typo and doc fixes
* Removing unused variable
 2017-12-13 14:08:11 +01:00
Jiri Pechanec
22dca4e498 DBZ-406 Buffering is disabled by default 2017-12-13 14:08:11 +01:00
Jiri Pechanec
5a5a85d920 DBZ-406 Support for look-ahead buffer with unlimited tx size 
Also incorporates DBZ-405
 2017-12-13 14:08:11 +01:00
Gunnar Morling
c88a521a52 DBZ-349 Avoiding need for map with overrides in config context 2017-12-13 12:34:30 +01:00
Andras Istvan Nagy
cc7459f4fb DBZ-349 removed jackson databind dependency originally introduced for DBZ-349 2017-12-13 12:34:30 +01:00
Andras Istvan Nagy
05a232d435 DBZ-349 refactoring to use a comma-separated list of tables in the snapshot.select.statement.overrides property and a separate config property for each override 2017-12-13 12:34:30 +01:00
Attila Szucs
c0d1c256ad DBZ-349 fix integration test 2017-12-13 12:34:30 +01:00
Andras Istvan Nagy
631c518d8e DBZ-349 Better support for large append-only tables by making the snapshotting process restartable 2017-12-13 12:34:30 +01:00
Jiri Pechanec
68f0a96a08 DBZ-461 Changed default bigint conversion to long 2017-12-02 21:04:29 +01:00
Gunnar Morling
db61ad2980 DBZ-491 Asserting offsets as per the given time and timezone (instead of now) 2017-12-01 10:55:43 +01:00
Gunnar Morling
79331608ae DBZ-342 Adding regression test for existing "adaptive" time handling mode 2017-11-30 09:07:30 +01:00
rkerner
c7ac481c43 [DBZ-342] fix broken MySQL data type "TIME" handling 2017-11-29 20:34:12 +01:00
Peter Goransson
f4d6af9429 DBZ-491 Normalizing date to UTC before asserting 2017-11-29 08:56:13 +01:00
Gunnar Morling
08310f9ea5 DBZ-483 Removing two unused methods from MySqlSchema 2017-11-24 17:04:01 +01:00
Jiri Pechanec
20a2cdfdea DBZ-476 Doubled quotes are parsed as escaped 2017-11-23 14:51:51 +01:00
Gunnar Morling
cc2afb3419 DBZ-473 Preventing unbounded polling in case connector is stopped during snapshotting 2017-11-21 11:24:05 +01:00
Jiri Pechanec
a1e0553b4d DBZ-475 Support for RENAME USER statement 2017-11-21 08:43:44 +01:00
Gunnar Morling
fe9222a633 DBZ-474 Adding Attila Szucs to COPYRIGHT.txt 2017-11-20 17:39:51 +01:00
Attila Szucs
07da3c8718 DBZ-474 Fix failure when COLLATE comes after NOT NULL 2017-11-20 17:39:47 +01:00
Gunnar Morling
6f9086d608 DBZ-217 Typo fix 2017-11-14 18:01:26 +01:00
Gunnar Morling
951abb5c62 DBZ-217 Making behavior for dealing with event deserialization failures configurable 2017-11-14 18:01:26 +01:00
Gunnar Morling
f705ae4ac6 DBZ-217 Handling exceptions ocurring during binlog event deserialization; 
* Catching the exception and wrapping it into a pseudo INCIDENT event
* Logging the exception and the current SourceInfo state
 2017-11-14 18:01:26 +01:00
Jiri Pechanec
55a252521f DBZ-447 Remove double initialization 2017-11-14 15:12:04 +01:00
Gunnar Morling
79a2e4a6c2 DBZ-396 Formatting and typo fix; 
Also adding Peter to COPYRIGHT.txt
 2017-11-14 11:37:28 +01:00
Peter Goransson
ba7478ffd7 DBZ-396 Adding BlockingReader to avoid high CPU utilization in 
"snapshot_only" mode;

No binlog readers will be set up in this mode, causing the idle snapshot
reader to be polled with high frequency after it has done its work.
Therefore a blocking reader is added in that case, which prevents the
polling.
 2017-11-14 11:32:27 +01:00
Jiri Pechanec
28258ece3f DBZ-260 Timestamp correctly handles timezones 2017-11-14 10:38:32 +01:00
Gunnar Morling
c4a29f6fb7 DBZ-437 Some clean-up 2017-11-14 09:40:54 +01:00
Jiri Pechanec
4d253d2987 DBZ-437 String tokens and SQL words are handled separately in procedure parsing 2017-11-14 09:40:54 +01:00
Jiri Pechanec
1611159a85 DBZ-415 Can parse embedded IFs, can parse IF with ( as first char 2017-11-14 09:00:58 +01:00
Jiri Pechanec
6db4e9b599 DBZ-425 Fix ALTER column for changing default value 2017-11-13 22:51:59 +01:00
Gunnar Morling
2744f54e9d DBZ-438 Renaming BufferedBlockingConsumer#flush() to close() to make clear it's onyl meant to be called once at the end 2017-11-13 14:19:39 +01:00
Jiri Pechanec
c905c7a558 DBZ-439 Support for PRECISION keyword 2017-11-13 13:58:53 +01:00
Gunnar Morling
5fbe742be8 DBZ-285 Specifying scope of dependencies in the individual POMs for the sake of comprehensibility 2017-11-10 16:48:32 +01:00
Jiri Pechanec
039566e717 DBZ-401 Log binlog reader position in case of exception 2017-11-03 09:12:09 +01:00
Jiri Pechanec
c1370556ff DBZ-428 TEXT datatype can have an optional length 2017-11-03 08:37:00 +01:00
Jiri Pechanec
e841cbd609 DBZ-429 Fixed signed default value handling 2017-11-03 06:13:05 +01:00
Jiri Pechanec
1fe753717f DBZ-412 Definition for column named column can be changed 2017-11-01 16:32:44 +01:00
Jiri Pechanec
decf822c65 DBZ-419 PK constraint name is optional 2017-10-30 17:31:08 +01:00
Gunnar Morling
ce16ef5eb3 DBZ-395 Reducing log level from info to debug 2017-10-26 11:31:55 +02:00
Jiri Pechanec
f93e1e9bcd DBZ-395 Make temp table regex stricter 2017-10-26 11:11:22 +02:00
Jiri Pechanec
1c6e652c71 DBZ-395 Filter out DROP TEMPORARY TABLE statements 2017-10-26 11:11:22 +02:00
Gunnar Morling
f86feaf9bd DBZ-408 Allowing for columns named "column" (escaped) in ALTER TABLE ADD/DROP... without COLUMN word 2017-10-26 10:20:25 +02:00
Gunnar Morling
1d7f41af26 DBZ-408 Consuming COLUMN token only for ALTER TABLE statements, so to support columns named "column" (escaped) in CREATE TABLE statements 2017-10-26 10:20:25 +02:00
Gunnar Morling
a38c4254f1 DBZ-402 Removing redundant call to initialize() 2017-10-19 21:43:35 +02:00
Randall Hauch
c71f6d2752 DBZ-402 Reader now has ability to initialize resource prior to starting 
Since some Reader implementations are combinations of other Readers, it is sometimes necessary for Reader implementations to initialize resources up front and not wait until they are called. Thus, a new `initialize()` method is added to Reader that will, for the combination readers, initialize all of the readers right away.

This makes it easier to correctly register JMX metrics, for example, up front once all of the readers have been configured for use, rather than to do so upon construction when it is not clear that a particular Reader will be used after it is constructed.
 2017-10-19 21:43:35 +02:00
Gunnar Morling
6a8e08db5d DBZ-363 Indentation fix; adding Ben Williams to copyright.txt 2017-10-18 10:31:00 +02:00
Ben Williams
a3b4fedd5f DBZ-363 Add support for BIGINT UNSIGNED handling for MySQL 2017-10-18 10:20:03 +02:00
Jiri Pechanec
875c3ac87a DBZ-336 Test for PAGE_CHECKSUM 2017-10-13 19:08:27 +02:00
Moira Tagle
4bcec0587a Fix inaccurate comment on MySQLSchema.tables method 2017-10-03 05:23:53 +02:00
Gunnar Morling
9958faee7b DBZ-378 Preventing NPE in MySqlConnectorTask#close() 2017-09-26 12:52:25 +02:00
Jenkins user
75937711fa [maven-release-plugin] prepare for next development iteration 2017-09-21 04:42:02 +00:00
Jenkins user
a89b9332e4 [maven-release-plugin] prepare release v0.6.0 2017-09-21 04:42:02 +00:00
LiuHanlin
4543c001b1 DBZ-359 Add support for dec and fixed for mysql ddl parser. 
According to Mysql document, https://dev.mysql.com/doc/refman/5.7/en/numeric-types.html
keywords DEC and FIXED are synonyms for DECIMAL.
 2017-09-19 09:12:09 +02:00
Gunnar Morling
99f39038bb DBZ-347 Formatting 2017-09-18 17:04:14 +02:00
Jiri Pechanec
38f8364647 DBZ-347 Simplified exception rethrowing 2017-09-18 17:02:21 +02:00
Jiri Pechanec
61ccefcc5c DBZ-347 Configurable behaviour for unknown DDL, failing by default 2017-09-18 17:02:21 +02:00
Jiri Pechanec
1c5326e731 DBZ-304 MySQL test conversion to dynamic database creation 2017-09-18 16:23:55 +02:00
Jiri Pechanec
7b9c29efe6 DBZ-210 Lock tables before starting tx in snapshot 2017-09-15 12:20:35 +02:00
Jiri Pechanec
81edc3d7ee DBZ-333 Filter database.history props out of JDBC URL 2017-09-15 11:45:53 +02:00
Jiri Pechanec
16671152f9 DBZ-346 Correctly parse MySQL merge tables 2017-09-15 10:03:44 +02:00
Jiri Pechanec
6fad4c62aa Added log statements for error events 2017-08-31 06:37:33 +02:00
Jiri Pechanec
8cb681f106 DBZ-219 Stop BinlogReaderClient background jobs 2017-08-29 14:55:04 +02:00
Jenkins user
214696ef0c [maven-release-plugin] prepare for next development iteration 2017-08-17 11:51:05 +00:00
Jenkins user
c867e6fea6 [maven-release-plugin] prepare release v0.5.2 2017-08-17 11:51:05 +00:00
Satyajit Vegesna
b4a0ac0c35 DBZ-324 Support for PAGE_CHECKSUM=1 2017-08-17 08:56:53 +02:00
Gunnar Morling
4d90fb5154 DBZ-284 Removing some trailing whitespace 2017-06-13 17:10:29 +02:00
Omar Al-Safi
107a3a8b56 DBZ-284 Fix NPE when inserting a NULL in a POINT column 2017-06-13 15:33:37 +02:00
Gunnar Morling
a8d1817c22 [maven-release-plugin] prepare for next development iteration 2017-06-09 16:14:31 +00:00
Gunnar Morling
3f512aace7 [maven-release-plugin] prepare release v0.5.1 2017-06-09 16:14:31 +00:00
Gunnar Morling
46bc2ab38b DBZ-228 Avoiding some auto-boxing; extracting some additions to initialization time 2017-06-09 13:53:45 +02:00
Omar Al-Safi
730767f6c5 DBZ-228 Fix formatting issue 2017-06-09 13:53:45 +02:00
Omar Al-Safi
451bd95a75 DBZ-228 Add UNSIGNED BIGINT support 2017-06-09 13:53:45 +02:00
Omar Al-Safi
5d86ae592f DBZ-228 Fix MySQL connector unsigned integer types 2017-06-09 11:52:14 +02:00
Randall Hauch
9581140c36 DBZ-216 MySQL connector should ignore DELETE FROM statements 
Parse and ignore any `DELETE` statements that might be seen in the binlog.

Theoretically, the binlog of a properly-configured MySQL server with row-level binlog enabled should never see these statements. However, users on Amazon RDS run into this quite frequently, and we should just handle and ignore them.
 2017-06-01 20:04:45 +02:00
Gunnar Morling
68fc0966a2 Fixing wrong display name 2017-06-01 14:52:33 +02:00
Gunnar Morling
2a22f3d4b0 DBZ-254 Right-padding values for fixed-length BINARY columns with 0x00 (zero byte) characters for MySQL; 
Also fixing JDBC types for binary data types for MySQL.
 2017-05-30 15:04:38 +02:00
Gunnar Morling
559fb1f600 DBZ-262 Passing enum constants to configuration in one more test 2017-05-30 11:18:56 +02:00
Gunnar Morling
f17ced3b4a DBZ-262 Letting configuration enum types implement EnumeratedValue 2017-05-23 21:26:22 +02:00
Gunnar Morling
c6674f8a4a DBZ-253 Adding integration test 2017-05-23 21:26:01 +02:00
Gunnar Morling
1903b7dfe2 DBZ-253 Avoiding exception when receiving DDL events with table maintenance statements 2017-05-23 21:26:01 +02:00
Gunnar Morling
40cb3e2530 DBZ-215 Validating configuration to prevent that database history topic and schema change topic collide 2017-05-22 08:21:22 +02:00
Randall Hauch
02e655fa53 DBZ-198 Add another DDL parser test case 
Added an additional test that is unable to reproduce the problem reported on April 4.
 2017-05-19 15:50:03 +02:00
Gunnar Morling
03ca03c90c DBZ-232 Removing unused constant 2017-05-19 09:28:56 +02:00
Randall Hauch
787959c4d0 DBZ-232 Removed the database and table recommenders 
It’s not clear how valuable these recommenders actually are. First, it’s not clear about the expected semantics: can the user use values that don’t appear in the recommended values? Second, the recommenders that return large numbers of values can be slow and can result in very large REST API responses.

Debezium was using recommenders to return the database and table/collection names, but these lists can be very large for large databases. Rather than cap the number of recommended values and have the recommender return a subset of all potential values, we will instead remove the recommenders altogether.
 2017-05-19 09:24:07 +02:00
Randall Hauch
8f69836c9e DBZ-232 Changed the database and table recommenders to return no more than 100 values 2017-05-19 09:24:07 +02:00
Gunnar Morling
37529f27b1 DBZ-243 Removing superfluous default value implementation for server name as the filed is mandatory 2017-05-19 09:14:53 +02:00
Randall Hauch
c10c1204f3 Merge pull request #230 from jpechane/retries 
DBZ-251 Prepopulate readbinlog_test database
 2017-05-18 16:58:53 -05:00
Randall Hauch
5fb878137e Merge pull request #223 from gunnarmorling/DBZ-243 
DBZ-243 Don't mention default value for "database.server.name"
 2017-05-18 12:09:39 -05:00
Jiri Pechanec
e79268ec90 Prepopulate readbinlog_test database 2017-05-12 07:20:53 +02:00
Randall Hauch
afef753d36 DBZ-242 Corrected MySQL filters to handle built-in tables 
When the `table.ignore.built` is set to `false`, the MySQL connector included all of the ~250 system tables yet the database and table filters were never applied to these system tables and therefore all would be captured. With this change, the database filters and table filters now apply to the system tables should they not be ignored.

Note that this does change the behavior of the connector when `table.ignore.built` is set to `false`. However, it is unlikely that anyone is seriously capturing all of the system table changes, so correcting the behavior is the preferred solution.
 2017-05-11 11:57:08 +02:00
Gunnar Morling
3f55398bab DBZ-243 Don't mention default value for "database.server.name" for MySQL connector 2017-05-08 20:24:28 +02:00
Gunnar Morling
baa8119df8 Typo fix 2017-05-08 17:40:19 +02:00
Gunnar Morling
1074de4efa DBZ-222 Some more clean-up 2017-05-04 09:25:25 +02:00
Gunnar Morling
6eea4c9717 DBZ-222 Some typo fixes 2017-05-04 08:53:51 +02:00
Omar Al-Safi
3d92011277 DBZ-222 Typo cleanup and removed unnecessary assertion 2017-05-04 08:53:34 +02:00
Gunnar Morling
5630b61be6 DBZ-222 dependency clean-up 2017-05-04 08:53:05 +02:00
Omar Al-Safi
2a17748a95 DBZ-222 Readded readbinlog_test.product and readbinlog_test.purchased to MySqlConnectorIT 2017-05-04 08:53:05 +02:00
Omar Al-Safi
791545c5f4 DBZ-222 Added support for MySQL POINT type 2017-05-04 08:53:05 +02:00
Jiri Pechanec
51a1c7cd69 DBZ-229 - check all privilege records 2017-04-26 15:25:59 +02:00
Randall Hauch
709cd8f3fe [maven-release-plugin] prepare for next development iteration 2017-03-27 11:28:12 -05:00
Randall Hauch
2bc3d45954 [maven-release-plugin] prepare release v0.5.0 2017-03-27 11:28:11 -05:00
Randall Hauch
25adc3f642 Merge pull request #209 from rhauch/dbz-205 
DBZ-205 Corrected MySQL connector to handle 2-digit years
 2017-03-27 11:10:04 -05:00
Randall Hauch
81f62b6961 DBZ-205 Corrected MySQL connector to handle 2-digit years 
MySQL has special handling of 2-digit years that it deems are ambiguous, such as the year value `17` that is actually treated as `2017`. Apparently the 2-digit values are stored in MySQL and the interpretation is performed when the data is extracted, so therefore the connector needs to also perform this adjustment of the year values. This commit uses the JDK’s `TemporalAdjuster` interface and passes this down to the requisite temporal-related datatype handling code. The MySQL connector then provides its own `TemporalAdjuster` implementation that adjusts the year values via the excellend JDK `Temporal` methods.

A row in one of the MySQL test databases was changed to use a 2-digit year of `16` while the test method still checks that the year is still 2016`, verifying that the year value is properly adjusted.
 2017-03-27 10:58:21 -05:00
Randall Hauch
74fbb4a140 DBZ-198 Corrected MySQL DDL parser 
The parser now handles `BEGIN…END` blocks better by properly handling `IF()` functions that are not `IF…THEN…END IF` control blocks, and `CASE … WHEN … END CASE` control blocks.
 2017-03-24 13:12:46 -05:00
Sanjay Kr Singh
ac4575a13c update mysql-dbz-198.ddl with 3 new test cases for PROCEDURE and 2 test cases for FUNCTION 
From L49 - one DDL test case for PROCEDURE ,
From L278 - one DDL test case for FUNCTION ,
From L433 - one more DDL test case for PROCEDURE with CURSOR,
From L713 -  ONE more DDL TEST FOR PROCEDURE,
From L755 - ONE more DDL TEST CASE FOR FUNCTION
 2017-03-24 12:29:17 +05:30
Randall Hauch
430d756062 [maven-release-plugin] prepare for next development iteration 2017-03-17 15:41:58 -05:00
Randall Hauch
536cbf6300 [maven-release-plugin] prepare release v0.4.1 2017-03-17 15:41:57 -05:00
Randall Hauch
2086b779b1 DBZ-204 Added test case but unable replicate error 
Added a test case that uses the MySQL DDL parser to parse similar DDL statements to those reported in the issue, but these are properly handled with the current state of the `master` branch.
 2017-03-17 14:15:25 -05:00
Randall Hauch
e5ee3847dd DBZ-195 Added tests to try to replicate a reported issue 
Added a table and inserted rows that tries to replicate the problem reported in DBZ-195, but the test was unable to replicate the problem. In fact, this really is no different than existing tests. Changed the log messages so that if/when this happens again it will be possible to know which row was problematic.
 2017-03-17 10:47:32 -05:00
Randall Hauch
ddbc1e07aa DBZ-197 Corrected MySQL connector to handle invalid enum values 
MySQL represents an invalid enum literal in the binlog events as an empty string or an value of `0`. Now, when the connector comes across such a value in the binlog, it will instead use an empty string for the enum literal.
 2017-03-16 13:55:24 -05:00
Randall Hauch
cf5391482a DBZ-198 Improved MySQL DDL parser to better handle blocks 
The MySQL parser now properly handles control blocks such as `BEGIN…END`, `IF…END IF`, `REPEAT…END REPEAT`, and `LOOP…END LOOP`, even in cases where the block is preceded by and terminated by a label.
 2017-03-16 13:32:21 -05:00
Randall Hauch
06d9bcb092 Merge pull request #199 from dasl-/daylightsavings 
DBZ-202 Fix daylight savings test issues
 2017-03-15 15:41:41 -05:00
dleibovic
782ccc66e5 fix daylight savings test issues 2017-03-15 15:44:03 -04:00
rich
06a25f8fd9 modify the query used to fetch the table list so that it includes tables only(no views) 2017-03-15 13:56:52 -04:00
Randall Hauch
b48ccce4b5 DBZ-200 Corrected MySQL DDL parser to better handle column definitions 
Apparently not all reserved words must be quoted when using them as colum names, so refactored MySQL’s DDL parser to better handle a variety of unquoted colum names that are reserved words.
 2017-03-08 12:12:27 -06:00
Josh Stanfield
c794416684 update to allow savepoint in mysql replication stream 2017-03-08 09:28:10 -07:00
dleibovic
9fd2afc8b4 Increase the default mysql init timeout to 60 seconds for slower computers. Also paramterize it so that users can pass a custom value via 'mvn clean install -Dmysql.init.timeout=80000' for example 2017-02-23 13:34:14 -05:00
rich
9aa49736c8 DBZ-140 when locking individual tables, use a single statement with all the table names instead of issuing a statement per table which causes a MySQL error 2017-02-16 15:45:29 -05:00
Randall Hauch
043a2d2d92 DBZ-194 Improved MySQL connector’s built-in table filtering 
The MySQL connector’s built-in table filter now just filters out all tables within the known built-in databases, and does not check the names of the tables. Thus, the connector should no longer filter out tables in other databases that happen to have the same names as the tables in the built-in databases.
 2017-02-14 09:23:39 -06:00
Randall Hauch
af94fa8759 DBZ-193 MySQL DDL parser handles FULLTEXT index 
Corrected the MySQL DDL parser to correctly handle `FULLTEXT` indexes within a `CREATE TABLE` statement. The parser was incorrectly using `canConsume(…)` with a list of options instead of `canConsumeAnyOf(…)`.
 2017-02-10 15:49:20 -06:00
Randall Hauch
9a4a177004 DBZ-188 Corrected JavaDoc 2017-02-10 15:39:22 -06:00
dleibovic
aa50bfe71a DBZ-188: Allow a debezium mysql connector to filter production of DML events into kafka by the mysql UUID of the event 
With GTIDs enabled, each transaction in the binlog contains a GTID event, which gives us access to the GTID of the transaction. The GTID has the following format: source_id:transaction_id, where source_id is the UUID of the mysql server the transaction was written to.

I propose to allow a debezium instance to be configured with a UUID pattern to check against before producing DML events into Kafka. Debezium would produce a DML event into kafka if and only if the UUID in the event's GTID matches the pattern with which debezium was configured.

The configuration for the UUID patterns will make use of the existing gtid.source.includes and gtid.source.excludes options. The DML event filtering will only be performed if the new option gtid.source.filter.dml.events is true.
 2017-02-10 14:14:10 -05:00
Randall Hauch
d2986710a5 DBZ-188 More efficient GTID source filters for MySQL Connector 
Changed the GTID source filters in the MySQL connector to be far more efficient when the filters specify literal UUIDs rather than regex patterns. In these cases, the predicate just checks whether a supplied value is in a hash set, and no regular expression patterns are used.

The GTID source filters can still be a combination of UUID literals and regular expressions, and the predicate will use the best implementation for each. For example, if the filters include all UUID literals, then regular expressions will never be used.
 2017-02-10 11:34:24 -06:00
Randall Hauch
8c60c29883 [maven-release-plugin] prepare for next development iteration 2017-02-07 14:22:12 -06:00
Randall Hauch
20134286e9 [maven-release-plugin] prepare release v0.4.0 2017-02-07 14:22:11 -06:00
Randall Hauch
403fee1375 DBZ-185 MySQL’s database history now filters GTID sources 
Corrects how the MySQL connector reloads database history to take into account the included and excluded GTID sources. This only affects a connector configured to capture changes from _multiple_ MySQL database servers when GTID sources are explicitly excluded or included.
 2017-02-07 11:21:22 -06:00
Randall Hauch
bb0800ca3a DBZ-140 Improved locking logic to support RDS 
Improved the MySQL connector's logic to better handle Amazon RDS that does not allow giving user `SUPER` privileges. As before, the connector starts a transaction and attempts to get a global read lock via `FLUSH TABLES WITH READ LOCK` to prevent writes to the database so that the binlog position can be accurately read _and_ the table schemas can be read without interference from other clients. Once that is done, the connector releases the global read lock and continues in the same transaction to read all table rows. This means that our snapshot is consistent, but we maintain the global read lock for a very short period of time.

Amazon's RDS and Aurora are hosted MySQL instances that do not allow users to have the `SUPER` privilege, which means the user cannot get a global read lock. In this case, the connector detects this error, continues to read the database and table names (without any lock), and _then_ uses `FLUSH TABLES <tableName> WITH READ LOCK` on each table that satisfies the filters to prevent changes from other clients. The connector then reads the table schemas, reads _all_ table rows, commits the transaction, and _finally_ releases the table locks.

Therefore, there are two very different behaviors/requirements when the user can't obtain a global read lock because of lack of privilege, like on RDS:

# The RDS user that the connector makes use of must also have the `LOCK TABLES` privilege; without it the connector will fail during the snapshot.
# The connector must hold the table read locks _until it has completed reading all of the tables_, since release the table locks using `UNLOCK TABLES` would prematurely commit our transaction and prevent us from getting a consistent snapshot. From the [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/flush.html):
> `UNLOCK TABLES` implicitly commits any active transaction only if any tables currently have been locked with `LOCK TABLES`. The commit does not occur for `UNLOCK TABLES` following `FLUSH TABLES WITH READ LOCK` because the latter statement does not acquire table locks.
 2017-02-06 13:56:55 -06:00
Randall Hauch
5490842449 Merge pull request #175 from rhauch/dbz-176 
DBZ-176 Corrected MySQL DDL parser to support creating triggers with definers
 2017-02-02 13:59:01 -06:00
Randall Hauch
74e5ba6448 DBZ-176 Corrected MySQL DDL parser to support creating triggers with definers 
The MySQL DDL parser was not correclty handling `DEFINER` clauses within `CREATE TRIGGER` or `CREATE EVENT` statements. Support for `DEFINER` clauses was recently added for the various forms of `CREATE PROCEDURE`, `CREATE FUNCTION` and `CREATE VIEW` statements. These are the only kinds of statements that have the definer attribute, per the [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/stored-programs-security.html).
 2017-02-02 12:44:28 -06:00
Randall Hauch
32a88fdc6f DBZ-184 Added database and table name to change event metadata 2017-02-02 12:09:53 -06:00
Randall Hauch
6230cab90e Merge pull request #173 from rhauch/dbz-113 
DBZ-113 Added MySQL threads to the event’s source metadata
 2017-02-02 12:00:19 -06:00
Randall Hauch
fe17b246af DBZ-113 Added MySQL threads to the event’s source metadata 
Changed the events’ `source` structure to optionally contain the identifier of the MySQL thread where appropriate. The thread is included on each `BEGIN` binlog event, so these are captured and added to all of the associated change events produced for that transaction.
 2017-02-02 11:53:32 -06:00
Randall Hauch
f2a65d03df DBZ-174 Added support for new binlog events 
MySQL recently added additional binlog events, and this commit adds support to handle these new events by ignoring them.
 2017-02-01 15:26:28 -06:00
Horia Chiorean
031c4a1552 DBZ-183 Fixes the BinlogReader's handling of TIMESTAMP columns to correctly account for timezones 2017-01-25 16:39:36 +02:00
Randall Hauch
a73f85a80f Merge pull request #162 from rareddy/DBZ-177 
DBZ-177: Providing an alternative way to create JDBC connection based …
 2017-01-13 13:37:38 -06:00
Ramesh Reddy
a9aace3480 DBZ-177: Providing an alternative way to create JDBC connection based on the configured JDBC driver class name and supplied classloader. The loading/creating the JDBC connections is not reliable when driver libraries in a different classloader than the DriverManager. 2017-01-13 12:58:14 -06:00
Horia Chiorean
a300d3e1cf DBZ-3 Changes the configuration of the Docker Maven plugin to only use alias naming when necessary and moves the PG connector ahead of the Mongo connector in the build 2016-12-27 14:44:33 +02:00
Horia Chiorean
23e3f59fa1 DBZ-3 Implements a connector for streaming changes from a Postgres database 
The version of the DB server required for this to work is at least 9.4
The commit also updates the general DBZ build system for:
* custom checkstyle package exclusions - required by the Postgres driver the protobuf code for now
* adds support for debugging Surefire and Failsafe
 2016-12-27 14:44:32 +02:00
Randall Hauch
e60839e76b DBZ-164 Improved MySQL snapshot reader logic 
Added more logic to the snapshot reader to better handle errors when reading the list of table names in each database. Now, any errors with a single database (e.g., some of the not-quite-a-database names described in the JIRA issue) will cause the snapshot reader to simply skip that database name and continue on (with proper logging).

This change also quotes all of the database and table names when used in SQL statements.
 2016-12-20 22:03:46 -06:00
Randall Hauch
fd7e152852 Merge pull request #142 from rhauch/dbz-151 
DBZ-151 Added new integration test framework
 2016-12-20 17:53:16 -06:00
Randall Hauch
ab1140ef70 Merge pull request #155 from rhauch/dbz-169 
DBZ-169 MySQL connector support for ON UPDATE clauses
 2016-12-20 17:48:06 -06:00
Randall Hauch
fe44380d4c Merge pull request #154 from rhauch/dbz-168 
DBZ-168 MySQL connector ignores XA binlog events
 2016-12-20 17:47:57 -06:00
Randall Hauch
a9a84cb6aa DBZ-152 Enabled MySQL connector to skip table count checks during snapshot 
Change the MySQL connector’s `min.row.count.to.stream.results` configuration property to accept a value of 0, which signifies that all `SELECT COUNT(*) FROM tableA` queries should be skipped and instead all results should be streamed.
 2016-12-20 17:40:57 -06:00
Randall Hauch
046702d959 DBZ-169 MySQL connector support for ON UPDATE clauses 
Corrected the MySQL DDL parser to support `ON UPDATE NOW()` clauses in addition to `ON UPDATE CURRENT_TIMESTAMP`.
 2016-12-20 16:19:18 -06:00
Randall Hauch
09f87cf190 DBZ-168 MySQL connector ignores XA binlog events 
MySQL 5.7.7 introduced new behavior for handling XA events in the binlog. See the [MySQL documentation|http://dev.mysql.com/doc/refman/5.7/en/xa-restrictions.html] for details. This PR changes the binlog reader so that `XA …` statements appearing in the binlog are ignored altogether.
 2016-12-20 15:32:44 -06:00
Randall Hauch
5dceb05f69 DBZ-151 Additional changes to improve test framework and MySQL integration tests 2016-12-20 10:58:56 -06:00
Randall Hauch
08e32a4a8b DBZ-151 Added multiple integration test modules to test various MySQL versions and configurations. 
These new modules run during the '-Passembly' profile and use the new integration test framework that compares all
output produced by a connector to expected results that were previously recorded and verified. These integration test modules
can be run manually with a simple build of those modules or their parent; only the top-level 'integration-tests' module is run
during the assembly profile during builds of the entire codebase.
 2016-12-20 09:18:10 -06:00
Randall Hauch
0851d8280c DBZ-166 Corrected shutdown logic of MySQL connector 
The MySQL connector uses several threads, so previously upon connector shutdown these threads were simply cancelled. This is fine for the binlog reader (which can stop at any moment), but is a poor approach for the snapshot as we didn’t always properly release the database resources and also didn’t complete the writing of the DDL history.

With this change, the snapshot reader stops in a very controlled manner, basically by having the 10-step snapshot procedure frequently check whether the reader is to continue working, and to completely avoid thread interruption altogether. And, the snapshot procedure will always clean up its database resources (locks, transactions, etc.), even if the procedure is stopped before completion.

This change also refactors how the snapshot and binlog reader are managed. This is no longer done in the MySqlConnectorTask class (which is busy enough), but rather the logic has been encapsulated in a new `ChainedReader` that makes use of a new `Reader` interface. This makes testing of `ChainedReader` easier, and ensure that `ChainedReader` relies only upon the primary methods of `Reader` rather than upon `AbstractReader`. `ChainedReader` handles multiple readers generically, and ensures that when stopped the readers are all handled correctly and completely process all records, yet avoid accidentally starting a subsequent reader(s) when stopping the previous reader.
 2016-12-15 10:55:18 -06:00
Randall Hauch
e3e66bf960 DBZ-161 Corrected MySQL connector logic when no GTIDs are used 
Corrected the logic of the MySQL connector when getting the server’s GTID set. Previously, this logic failed if GTIDs are not used.
 2016-12-08 08:09:52 -06:00
Dennis Persson
acd7bd8fa5 DBZ-142 Handle national character set columns in DDL parser 2016-12-07 07:38:30 +01:00
Randall Hauch
c762a221b7 DBZ-162 Corrected DDL parsing of MySQL functions 
The MySQL DDL parser was not properly consuming function declarations. For functions, the parser consumes the entire statement without handline the various expressions within the function declaration, but the parser was not properly finding the end of the statement and instead was continuing to try to consume values beyond the end of the statement.

Specifically, when the parser consumes a `BEGIN`, it looks for a corresponding `END`. However, if it encountered an `END IF`, the `IF` plus any remaining tokens were left on the token stream and unprocessed. This confused the parser, which keep looking for statements and ultimately ended with a `No more content` error.

This case was replicated in integration tests, and the code fixed to properly find the end of the statements.
 2016-12-06 17:34:52 -06:00
Randall Hauch
c72242eeb0 Merge pull request #145 from sherafpm/bugfix/DBZ-160 
DBZ-160 - Issue while parsing create table script with ENUM type and default value 'b'
 2016-12-06 14:21:23 -06:00
Randall Hauch
eedc4fba00 DBZ-163 Corrected assembly profile in build 
The Travis-CI builds run the Maven build using the `assembly` profile, and this has been failing quite a bit lately.

The first problem appears to be that the Travis-CI environment recently changed to have port 3306 taken, which means that our build fails to start any Docker containers for MySQL that attempt to use this port. A simple fix is to use different ports for the assembly build.

However, trying to change the port numbers for some of the profiles caused a lot of problems, and to correct these required refactoring how the properties are set. The Docker Maven plugin is now configured with separate properties that are set once (depending upon the profile) to determine the port assignments of the various Docker containers. The Failsafe plugin executions then use these Maven properties when setting the system variables (e.g., `database.host`) needed in the integration tests. This appears to have worked, but it still is a bit fragile. For example, the assembly profile defines several Failsafe executions, and during this profile these should be the only executions run; however, if not all the properties are set properly, the build seems to also run the default Failsafe execution in addition to the other `assembly` profile executions. (I think properties can’t only be defined in the execution, but need to also be defined in the Failsafe configuration.)

The “alternative” MySQL Docker images were removed, since they basically should not provide any different behavior than the `mysql/mysql-server` images we normally used. The extra containers required a lot more resources to run and dramatically increased the complexity of the build.

A few other trivial changes were made.
 2016-12-05 16:37:59 -06:00
Randall Hauch
2b2bf693d7 DBZ-163 Changed Travis-CI build to skip the install dependencies step 2016-12-02 15:43:57 -06:00
Sherafudheen PM
ee52219736 DBZ-160 - Issue while parsing create table script with ENUM type and default value 'b' 2016-12-02 17:42:44 +05:30
Randall Hauch
0bf3b4c9f3 DBZ-157 Upgraded Docker Maven plugin 
Upgraded the Docker Maven plugin to 0.18.1, which required changing our use of the `docker.image` to `docker.filter` (per the [changes in 0.17.1](https://github.com/fabric8io/docker-maven-plugin/blob/master/doc/changelog.md)).
 2016-11-22 09:23:07 -06:00
Randall Hauch
a82ae5691b Reduce the log verbosity of the MySQL tests 2016-11-14 13:41:10 -06:00
Randall Hauch
d80bc1bfd7 DBZ-153 MySQL connector supports enum and set values with parentheses 
Changed the MySQL connector to support ENUM and SET literals with parentheses.
 2016-11-14 12:22:08 -06:00
Randall Hauch
8a52cda0dc DBZ-150 Changed the order of events when a row's key is changed. 2016-11-09 14:42:43 -06:00