Merge pull request #14 from rhauch/dbz-14

DBZ-14 Corrected the 'alt-mysql' Maven profile
This commit is contained in:
Randall Hauch 2016-02-16 16:53:53 -06:00
commit 997456d027
5 changed files with 217 additions and 167 deletions

View File

@ -2,17 +2,13 @@ This module builds a custom Docker image based upon the [mysql/mysql-server:5.7]
The [mysql](https://hub.docker.com/r/_/mysql/) images are maintained by Docker, and provide a more complete installation with all of the development tools (including the `mysqlbinlog` utility). Startup is a lot more verbose, and only with recent images could our build easily discover when the server was finally ready (since the server is started and stopped several times).
## Starting MySQL Server
## Using MySQL Server
As mentioned in the [README.md]() file, our Maven build can be used to start a container using either one of these images. By default, the `mysql/mysql-server:5.7` image is used:
As mentioned in the [README.md]() file, our Maven build can be used to start a container using either one of these images. The `mysql/mysql-server:5.7` image is used:
$ mvn docker:start
but the `mysql:5.7` image can be used by using the `alt-server` profile:
$ mvn docker:start -Palt-server
Both commands leave the container running so that you can use the running MySQL server. For example, you can establish a `bash` shell inside the container (named `database`) by using Docker in another terminal:
The command leaves the container running so that you can use the running MySQL server. For example, you can establish a `bash` shell inside the container (named `database`) by using Docker in another terminal:
$ docker exec -it database bash
@ -27,11 +23,30 @@ To stop and remove the `database` container, simply use the following commands:
$ docker stop database
$ docker rm database
or
$ mvn docker:stop
## Using the alternative MySQL Server
To use the more complete installation of MySQL found in the `mysql:5.7` Docker image, use the `alt-server` Maven profile when starting or stopping the Docker container:
$ mvn docker:start -Palt-server
and
$ mvn docker:stop -Palt-server
All other functionality remains exactly the same, including the ability to run the build using this Docker image:
$ mvn clean install -Palt-server
## Using Docker directly
Although using the Maven command is far simpler, the Maven command for the `alt-server` profile really just runs (via the Jolokia Maven plugin) a Docker command to start the container, so it's equivalent to:
$ docker run -it --name database -p 3306:3306 -v $(pwd)/src/test/docker/sample:/etc/mysql/conf.d -v $(pwd)/src/test/docker/init:/docker-entrypoint-initdb.d -e MYSQL_DATABASE=mysql -e MYSQL_ROOT_PASSWORD=debezium-rocks -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7
$ docker run -it --name database -p 3306:3306 -v $(pwd)/src/test/docker/alt-server:/etc/mysql/conf.d -v $(pwd)/src/test/docker/init:/docker-entrypoint-initdb.d -e MYSQL_DATABASE=mysql -e MYSQL_ROOT_PASSWORD=debezium-rocks -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7
This will use the `mysql:5.7` image to start a new container named `database` where the MySQL instance uses the combined startup settings from `/etc/mysql/my.cnf` (defined in the Docker image) and the same local configuration file we used in the integration test MySQL container, `src/test/docker/mysql.cnf` (mounted into the container at `/etc/mysql/conf.d/mysql.cnf`). The settings from the latter file take precedence.

View File

@ -14,15 +14,13 @@ The MySQL connector can also be used as a library without Kafka or Kafka Connect
## Testing
This module contains both unit tests and integration tests.
A *unit test* is a JUnit test class named `*Test.java` or `Test*.java` that never requires or uses external services, though it can use the file system and can run any components within the same JVM process. They should run very quickly, be independent of each other, and clean up after itself.
An *integration test* is a JUnit test class named `*IT.java` or `IT*.java` that uses a MySQL database server running in a custom Docker container. The build will automatically start the MySQL container before the integration tests are run and automatically stop and remove it after all of the integration tests complete (regardless of whether they suceed or fail). All databases used in the integration tests are defined and populated using `*.sql` files and `*.sh` scripts in the `src/test/docker/init` directory, which are copied into the Docker image and run in lexicographical order by MySQL upon startup. Multiple test methods within a single integration test class can reuse the same database, but generally each integration test class should use its own dedicated database(s).
An *integration test* is a JUnit test class named `*IT.java` or `IT*.java` that uses a MySQL database server running in a custom Docker container based upon the [mysql/mysql-server:5.7](https://hub.docker.com/r/mysql/mysql-server/) Docker image maintained by the MySQL team. The build will automatically start the MySQL container before the integration tests are run and automatically stop and remove it after all of the integration tests complete (regardless of whether they suceed or fail). All databases used in the integration tests are defined and populated using `*.sql` files and `*.sh` scripts in the `src/test/docker/init` directory, which are copied into the Docker image and run in lexicographical order by MySQL upon startup. Multiple test methods within a single integration test class can reuse the same database, but generally each integration test class should use its own dedicated database(s).
Running `mvn install` will compile all code and run the unit tests. If there are any compile problems or any of the unit tests fail, the build will stop immediately. Otherwise, the command will continue to create the module's artifacts, create the Docker image with MySQL and custom scripts, start the Docker container, run the integration tests, stop the container (even if there are integration test failures), and run checkstyle on the code. If there are still no problems, the build will then install the module's artifacts into the local Maven repository.
@ -46,13 +44,7 @@ If you want to debug integration tests by stepping through them in your IDE, usi
$ mvn docker:start
will start the default MySQL container based upon the [mysql/mysql-server:5.7](https://hub.docker.com/r/mysql/mysql-server/) Docker image maintained by the MySQL team, or
$ mvn docker:start -Palt-mysql
to start the MySQL container named based upon the Docker-maintained [mysql](https://hub.docker.com/r/_/mysql/) Docker image maintained by Docker (the company). (The former is used by default because it starts a bit faster, though it doesn't have a full installation of MySQL. The latter is a full-installation whose startup is a bit slower and more verbose, and it includes tools such as `mysqldump` and `mysqlbinlog`. Both containers are named `database`, configured identically, and initialized with the same databases and content, so they should behave the same for the integration tests.)
Again, the container and database server will be initialized as usual but will continue to run. Now you can use your IDE to run/debug one or more integration tests. Just be sure that the integration tests clean up their database before (and after) each test, and that you run the tests with VM arguments that define the required system properties, including:
will start the default MySQL container and run the database server. Now you can use your IDE to run/debug one or more integration tests. Just be sure that the integration tests clean up their database before (and after) each test, and that you run the tests with VM arguments that define the required system properties, including:
* `database.dbname` - the name of the database that your integration test will use; there is no default
* `database.hostname` - the IP address or name of the host where the Docker container is running; defaults to `localhost` which is likely for Linux, but on OS X and Windows Docker it will have to be set to the IP address of the VM that runs Docker (which you can find by looking at the `DOCKER_HOST` environment variable).
@ -66,6 +58,10 @@ For example, you can define these properties by passing these arguments to the J
When you are finished running the integration tests from your IDE, you have to stop and remove the Docker container (conveniently named "database") before you can run the next build:
$ mvn docker:stop
or using Docker directly:
$ docker stop database
$ docker rm database
@ -77,8 +73,32 @@ Sometimes you may want to inspect the state of the database(s) after one or more
$ mvn integration-test
### Stopping the Docker container
This instructs Maven to run the normal Maven lifecycle through `integration-test`, and to stop before the `post-integration-test` phase when the Docker container is normally shut down and removed. Be aware that you will need to manually stop and remove the container (conveniently named "database") before running the build again:
$ mvn docker:stop
or using Docker directly:
$ docker stop database
$ docker rm database
### Using an alternative MySQL Server
All of the above commands will start the MySQL Docker container that is built upon the [mysql/mysql-server:5.7](https://hub.docker.com/r/mysql/mysql-server/) Docker image maintained by the MySQL team. This image has an "optimized" MySQL server that includes only a portion of the full installation (e.g., it excludes some tools such as `mysqlbinlog`). However, it starts a little faster and is less verbose in its output.
This module defines the `alt-mysql` Maven profile that will instead use the [mysql](https://hub.docker.com/r/_/mysql/) Docker image maintained by Docker (the company). This is a bit more verbose, but it includes all of the MySQL utilities, including `mysqlbinlog` that may be necessary to properly debug and analyze the behavior of the integration tests.
To use the alternative Docker image, simply specify the `alt-mysql` Maven profile on all of the Maven commands, including a build:
$ mvn clean install -Palt-mysql
or to manually start the Docker container and keep it running:
$ mvn docker:start -Palt-mysql
or to stop and remove the Docker container:
$ mvn docker:start -Palt-mysql

View File

@ -81,11 +81,116 @@
<database.port>3306</database.port>
<database.user>mysqluser</database.user>
<database.password>mysqlpw</database.password>
<!--
By default, we should use the default Docker image. This propery is changed with different profiles.
-->
<docker.image>debezium-mysql57-server</docker.image>
<docker.skip>false</docker.skip>
</properties>
<build>
<plugins>
<!-- Build a Docker image of our MySQL installation, and run it as a container for our integration tests -->
<plugin>
<groupId>org.jolokia</groupId>
<artifactId>docker-maven-plugin</artifactId>
<configuration>
<watchInterval>500</watchInterval>
<logDate>default</logDate>
<verbose>true</verbose>
<autoPull>always</autoPull>
<images>
<image>
<!-- A custom Docker image built on top of an "optimized" MySQL installation. -->
<name>debezium-mysql57-server</name>
<alias>database</alias>
<build>
<assembly>
<basedir>/docker-entrypoint-initdb.d</basedir>
<dockerFileDir>${project.basedir}/src/test/docker</dockerFileDir>
</assembly>
</build>
<run>
<namingStrategy>alias</namingStrategy>
<env>
<MYSQL_ROOT_PASSWORD>debezium-rocks</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>mysql</MYSQL_DATABASE> <!-- database created upon init -->
<MYSQL_USER>${database.user}</MYSQL_USER>
<MYSQL_PASSWORD>${database.password}</MYSQL_PASSWORD>
</env>
<ports>
<port>${database.port}:3306</port>
</ports>
<log>
<prefix>mysql</prefix>
<enabled>true</enabled>
<color>yellow</color>
</log>
<wait>
<log>MySQL init process done. Ready for start up.</log>
<time>30000</time> <!-- 30 seconds max -->
</wait>
</run>
</image>
<image>
<!-- A Docker image using a complete MySQL installation. -->
<name>mysql:5.7</name>
<alias>database</alias>
<run>
<namingStrategy>alias</namingStrategy>
<volumes>
<bind>
<volume>${project.basedir}/src/test/docker/alt-server:/etc/mysql/conf.d</volume>
<volume>${project.basedir}/src/test/docker/init:/docker-entrypoint-initdb.d</volume>
</bind>
</volumes>
<env>
<MYSQL_ROOT_PASSWORD>debezium-rocks</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>mysql</MYSQL_DATABASE> <!-- database created upon init -->
<MYSQL_USER>${database.user}</MYSQL_USER>
<MYSQL_PASSWORD>${database.password}</MYSQL_PASSWORD>
</env>
<ports>
<port>${database.port}:3306</port>
</ports>
<log>
<prefix>alt-mysql</prefix>
<enabled>true</enabled>
<color>yellow</color>
</log>
<wait>
<log>Version: \'5\.7\.11-log\' socket: \'/var/run/mysqld/mysqld\.sock\' port: 3306 MySQL Community Server \(GPL\)</log>
<time>30000</time>
</wait>
</run>
</image>
</images>
</configuration>
<!--
Connect this plugin to the maven lifecycle around the integration-test phase:
start the container in pre-integration-test and stop it in post-integration-test.
-->
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>build</goal>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<!--
Unlike surefire, the failsafe plugin ensures 'post-integration-test' phase always runs, even
when there are failed integration tests. We rely upon this to always shut down the Docker container
after the integration tests (defined as '*IT.java') are run.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
@ -138,164 +243,42 @@
</testResource>
</testResources>
</build>
<!--
Define several profiles for working with different Docker images (or no Docker whatsoever)
-->
<profiles>
<!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Do not perform any Docker-related functionality
To use, specify "-DskipITs" on the Maven command line.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
<profile>
<id>docker</id>
<id>skip-integration-tests</id>
<activation>
<activeByDefault>false</activeByDefault>
<property>
<name>!skipITs</name>
<name>skipITs</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.jolokia</groupId>
<artifactId>docker-maven-plugin</artifactId>
<configuration>
<watchInterval>500</watchInterval>
<logDate>default</logDate>
<verbose>true</verbose>
<autoPull>always</autoPull>
<images>
<image>
<name>debezium-mysql57-server</name>
<alias>database</alias>
<build>
<assembly>
<basedir>/docker-entrypoint-initdb.d</basedir>
<dockerFileDir>${project.basedir}/src/test/docker</dockerFileDir>
</assembly>
</build>
<run>
<namingStrategy>alias</namingStrategy>
<!--volumes>
<bind>
<volume>${project.build.directory}/data:/var/lib/postgresql/data</volume>
</bind>
</volumes-->
<env>
<MYSQL_ROOT_PASSWORD>debezium-rocks</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>mysql</MYSQL_DATABASE> <!-- database created upon init -->
<MYSQL_USER>${database.user}</MYSQL_USER>
<MYSQL_PASSWORD>${database.password}</MYSQL_PASSWORD>
</env>
<ports>
<port>${database.port}:3306</port>
</ports>
<log>
<prefix>mysql</prefix>
<enabled>true</enabled>
<color>yellow</color>
</log>
<wait>
<log>MySQL init process done. Ready for start up.</log>
<time>30000</time> <!-- 30 seconds max -->
</wait>
</run>
</image>
</images>
</configuration>
<!--
Connect this plugin to the maven lifecycle around the integration-test phase:
start the container in pre-integration-test and stop it in post-integration-test.
-->
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>build</goal>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<docker.skip>true</docker.skip>
</properties>
</profile>
<!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the alternative Docker image for MySQL.
To use, specify "-Dalt-mysql" or -Palt-mysql on the Maven command line.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
<profile>
<id>alt-mysql</id>
<activation>
<activeByDefault>false</activeByDefault>
<property>
<name>alt-mysql</name>
</property>
</activation>
<build>
<plugins>
<!-- Build a Docker image of our MySQL installation, and run it as a container for our integration tests -->
<plugin>
<groupId>org.jolokia</groupId>
<artifactId>docker-maven-plugin</artifactId>
<configuration>
<watchInterval>500</watchInterval>
<logDate>default</logDate>
<verbose>true</verbose>
<autoPull>always</autoPull>
<images>
<image>
<name>mysql:5.7</name>
<alias>database</alias>
<run>
<namingStrategy>alias</namingStrategy>
<volumes>
<bind>
<volume>${project.basedir}/src/test/docker/alt-server:/etc/mysql/conf.ddata:/var/lib/postgresql/data</volume>
</bind>
<bind>
<volume>${project.basedir}/src/test/docker/init:/docker-entrypoint-initdb.d</volume>
</bind>
</volumes>
<env>
<MYSQL_ROOT_PASSWORD>debezium-rocks</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>mysql</MYSQL_DATABASE> <!-- database created upon init -->
<MYSQL_USER>${database.user}</MYSQL_USER>
<MYSQL_PASSWORD>${database.password}</MYSQL_PASSWORD>
</env>
<ports>
<port>${database.port}:3306</port>
</ports>
<log>
<prefix>mysql</prefix>
<enabled>true</enabled>
<color>yellow</color>
</log>
<wait>
<log>MySQL Community Server (GPL)</log>
<time>30000</time> <!-- 30 seconds max -->
</wait>
</run>
</image>
</images>
</configuration>
<!--
Connect this plugin to the maven lifecycle around the integration-test phase:
start the container in pre-integration-test and stop it in post-integration-test.
-->
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>build</goal>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<docker.image>mysql:5.7</docker.image>
<docker.skip>false</docker.skip>
</properties>
</profile>
</profiles>
</project>

View File

@ -2,6 +2,33 @@
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
skip-host-cache
skip-name-resolve
#datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
#secure-file-priv=/var/lib/mysql-files
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
#log-error=/var/log/mysqld.log
#pid-file=/var/run/mysqld/mysqld.pid
# ----------------------------------------------
# Debezium ingest

View File

@ -74,7 +74,7 @@
<version.war.plugin>2.5</version.war.plugin>
<version.codehaus.helper.plugin>1.8</version.codehaus.helper.plugin>
<version.google.formatter.plugin>0.3.1</version.google.formatter.plugin>
<version.docker.maven.plugin>0.13.6</version.docker.maven.plugin>
<version.docker.maven.plugin>0.13.9</version.docker.maven.plugin>
<!-- Dockerfiles -->
<docker.maintainer>Debezium community</docker.maintainer>
@ -88,7 +88,12 @@
<!--Skip long running tests by default-->
<skipLongRunningTests>true</skipLongRunningTests>
<!-- Don't skip integration tests by default -->
<skipITs>false</skipITs>
<!-- Run the 'default' docker image by default -->
<!--docker>default</docker-->
</properties>
<modules>
<module>support/checkstyle</module>