Commit Graph

203 Commits

Author SHA1 Message Date
Randall Hauch
dab0440612 DBZ-14 Corrected the 'alt-mysql' Maven profile so that it can be used with any of the other Maven commands. 2016-02-16 16:37:30 -06:00
Randall Hauch
c9e13f0668 Merge pull request #12 from christian-posta/ceposta-int-test-disable
add option to run without integration tests
2016-02-15 20:11:33 -06:00
Christian Posta
c730685a01 add option to run without integration tests 2016-02-15 16:26:32 -07:00
Randall Hauch
a4bbcecd7e Merge pull request #13 from rhauch/dbz-1
DBZ-1 Completed integration testing and debugging of the MySQL connector
2016-02-15 15:05:54 -06:00
Randall Hauch
73f3c9836b DBZ-1 Completed integration testing and debugging of the MySQL connector 2016-02-15 14:46:12 -06:00
Randall Hauch
e43b518577 Merge pull request #11 from rhauch/dbz-11
DBZ-11 Build can skip long-running unit and integration tests
2016-02-04 15:37:09 -06:00
Randall Hauch
1a59f9b07c DBZ-11 Build can skip long-running unit and integration tests 2016-02-04 15:35:27 -06:00
Randall Hauch
64187eb390 Merge pull request #10 from rhauch/dbz-10
DBZ-10 Added small utility so unit tests can run an embedded Kafka cluster in-process
2016-02-04 15:29:25 -06:00
Randall Hauch
54b822bb72 DBZ-10 Added small utility so unit tests can run an embedded Kafka cluster within the same process.
This utility is only suitable for unit tests and therefore is defined in the test JAR of the `debezium-core` module. It certainly should never be used for production purposes.
2016-02-04 15:18:27 -06:00
Randall Hauch
52a7c096a4 Merge pull request #9 from rhauch/dbz-9
DBZ-9 Added MySQL whitelist and blacklists on tables and databases.
2016-02-04 07:56:39 -06:00
Randall Hauch
c501f8486f DBZ-9 Added MySQL whitelist and blacklists on tables and databases. 2016-02-04 07:56:13 -06:00
Randall Hauch
89c17d6d16 Merge pull request #8 from rhauch/dbz-1
DBZ-1 Refactored the connector modules and improved the embedded engine framework
2016-02-03 16:12:05 -06:00
Randall Hauch
70fc601c0f DBZ-8 Added documentation about embedded engines. 2016-02-03 16:09:43 -06:00
Randall Hauch
fbae6d75c8 DBZ-1 Renamed EmbeddedConnector to EmbeddedEngine and improved README 2016-02-03 15:33:57 -06:00
Randall Hauch
37d6a5e7da DBZ-1 Expanded documentation and improved EmbeddedConnector framework
Changed the EmbeddedConnector framework to initialize all major components via configuration properties rather than through the public builder. This increases the size of the configurations, but it simplifies what embedding applications must do to obtain an EmbeddedConnector instance.

The DatabaseHistory framework was also changed to be configurable in similar ways to the OffsetBackingStore. Essentially, connectors that want to use it (like the MySqlConnector) will describe it as part of the connector's configuration, allowing more flexibility in which DatabaseHistory implementation is used and how it is configured whether in Kafka Connector or as part of the EmbeddedConnector.

Added a README.md to `debezium-embedded` to provide documentation and sample code showing how to use the EmbeddedConnector.
2016-02-03 14:11:53 -06:00
Randall Hauch
0e58dba9d6 DBZ-1 Renamed the connector modules and packages 2016-02-02 16:58:48 -06:00
Randall Hauch
44b5139ebf Merge pull request #7 from rhauch/dbz-7
DBZ-7 Require Docker 1.9 or higher rather than 1.6 or higher
2016-02-02 16:16:44 -06:00
Randall Hauch
ff25b4de0a DBZ-7 Require Docker 1.9 or higher rather than 1.6 or higher 2016-02-02 16:16:17 -06:00
Randall Hauch
baaf2ee6e9 Merge pull request #6 from rhauch/dbz-1
DBZ-1 Added the MySQL source connector
2016-02-02 15:02:58 -06:00
Randall Hauch
2da5b37f76 DBZ-1 Added support for recording and recovering database schema
Adds a small framework for recording the DDL operations on the schema state (e.g., Tables) as they are read and applied from the log, and when restarting the connector task to recover the accumulated schema state. Where and how the DDL operations are recorded is an abstraction called `DatabaseHistory`, with three options: in-memory (primarily for testing purposes), file-based (for embedded cases and perhaps standalone Kafka Connect uses), and Kafka (for normal Kafka Connect deployments).

The `DatabaseHistory` interface methods take several parameters that are used to construct a `SourceRecord`. The `SourceRecord` type was not used, however, since that would result in this interface (and potential extension mechanism) having a dependency on and exposing the Kafka API. Instead, the more general parameters are used to keep the API simple.

The `FileDatabaseHistory` and `MemoryDatabaseHistory` implementations are both fairly simple, but the `FileDatabaseHistory` relies upon representing each recorded change as a JSON document. This is simple, is easily written to files, allows for recovery of data from the raw file, etc. Although this was done initially using Jackson, the code to read and write the JSON documents required a lot of boilerplate. Instead, the `Document` framework developed during Debezium's very early prototype stages was brought back. It provides a very usable API for working with documents, including the ability to compare documents semantically (e.g., numeric values are converted to be able to compare their numeric values rather than just compare representations) and with or without field order.

The `KafkaDatabaseHistory` is a bit more complicated, since it uses a Kafka broker to record all database schema changes on a single topic with single partition, and then upon restart uses it to recover the history from the dedicated topics. This implementation also records the changes as JSON documents, keeping it simple and independent of the Kafka Connect converters.
2016-02-02 14:27:14 -06:00
Randall Hauch
6796fe32be DBZ-1 Added the initial stages of a MySQL source connector
The connector is in a basic working state, although it is not well tested yet and upon restart does not recover the schema state from the previous run.
2016-01-29 10:12:28 -06:00
Randall Hauch
71cfbb16f8 Removed unused README-project.md file 2016-01-27 09:10:24 -06:00
Randall Hauch
d617f7c02b Readded copyright file with correct case. 2016-01-27 09:09:30 -06:00
Randall Hauch
9009a8e7d1 Removed copyright file 2016-01-27 09:08:43 -06:00
Randall Hauch
457c480fa2 Renamed license file to mirror form used in other top-level filenames. 2016-01-27 09:06:23 -06:00
Randall Hauch
991d1869e1 Merge pull request #5 from rhauch/dbz-5
DBZ-5 Added change log file
2016-01-27 09:04:46 -06:00
Randall Hauch
39cb1eb8d5 Merge pull request #4 from rhauch/dbz-6
DBZ-6 Enforce line ending style for most file types
2016-01-27 09:02:40 -06:00
Randall Hauch
a9de015322 DBZ-5 Added change log file. 2016-01-27 09:01:44 -06:00
Randall Hauch
9b283f5792 DBZ-6 Enforce line ending style for most file types. 2016-01-27 08:55:09 -06:00
Randall Hauch
74f5b1ad4b Merge pull request #3 from rhauch/DBZ-4
DBZ-4 Changed copyright statement in source code headers and adjusted checkstyle rules
2016-01-27 08:44:35 -06:00
Randall Hauch
1ce6e1dde5 DBZ-4 Added copyright.txt file with generated list of contributors based upon Git history and Java @author tags. 2016-01-27 08:38:10 -06:00
Randall Hauch
d9090ed67b DBZ-4 Removed unused files, most of which were originally copied from the ModeShape codebase. 2016-01-27 08:37:23 -06:00
Randall Hauch
4c538d4e54 DBZ-4 Changed copyright statement in source code headers and adjusted checkstyle rules. 2016-01-27 08:12:01 -06:00
Randall Hauch
ddf61cec46 Merge pull request #2 from rhauch/update-documentation
Update documentation to reflect new copyright approach
2016-01-25 19:04:33 -06:00
Randall Hauch
eff1f665fa Updated checkstyle rule for headers, and corrected several incorrect headers. 2016-01-25 18:59:25 -06:00
Randall Hauch
a0a8953d2a Updated the copyright dates per new approach. 2016-01-25 18:33:08 -06:00
Randall Hauch
b7f2221107 Updated the README and added a CONTRIBUTE.md file with details for developers. 2016-01-25 18:33:07 -06:00
Randall Hauch
671172a6d3 Merge pull request #1 from rhauch/fix-travis
Changed Docker usage on Travis-CI
2016-01-25 18:31:30 -06:00
Randall Hauch
d7e52165be Changed MySQL binlog reading integration test to eliminate concurrent issues. 2016-01-25 18:19:33 -06:00
Randall Hauch
4ddd4b33be Changed Docker usage on Travis-CI 2016-01-25 16:12:07 -06:00
Randall Hauch
772977f391 Attempted to correct Travis build error 2016-01-25 13:50:17 -06:00
Randall Hauch
5e4c428285 Correct return type for function 2016-01-25 13:41:38 -06:00
Randall Hauch
ed36e25861 Added Travis build status image to README 2016-01-25 13:35:03 -06:00
Randall Hauch
8b31a54495 Added support for Travis-CI builds. 2016-01-25 13:31:50 -06:00
Randall Hauch
74ddfe1ec5 Updated MySQL integration test 2016-01-25 13:01:39 -06:00
Randall Hauch
71e90b5a69 Added MySQL ingest module with support for reading DDL statements. 2016-01-23 08:26:52 -06:00
Randall Hauch
8e6c615644 Added utilities for managing a relational schema's table definitions, with support for updating those by reading DDL 2016-01-20 08:53:29 -06:00
Randall Hauch
7b2a0927ce Updated readme files 2015-12-02 11:59:01 -06:00
Randall Hauch
dffdfd8049 Added debezium-core and MySQL binary log reading tests. 2015-11-24 15:54:37 -06:00
Randall Hauch
42926f17f3 Enabled binary replication log on MySQL config used in integration tests 2015-11-19 09:14:32 -06:00