Commit Graph

7146 Commits

Author SHA1 Message Date
Martin Medek
35bc00f642 DBZ-5165 created job-dsl for ocp system pipeline, renamed deployment parameters 2022-08-04 19:04:56 +02:00
Martin Medek
b7cbbb3bb2 DBZ-5165 refactoring the testsuite scripts 2022-08-04 19:04:56 +02:00
Martin Medek
7a265104bc DBZ-5165 added apicurio step and a jenkins job to build the testsuite image 2022-08-04 19:04:56 +02:00
Martin Medek
6ee07d6a65 DBZ-5165 added oracle project system property to pom, refactor. Added docker tag to DockerRhelMongoConnectorIT 2022-08-04 19:04:56 +02:00
Martin Medek
116e2cfc5b DBZ-5165 modified all db fixtures to not use loadbalancers when running in ocp 2022-08-04 19:04:56 +02:00
Martin Medek
57eeb46c20 DBZ-5165 removed usage of postgresql service inside ocp, changed deployment.yml to run postgresql tests, codestyle 2022-08-04 19:04:56 +02:00
Martin Medek
2bad919b9d DBZ-5165 added Postgresql loadBalancer (not)creation logic 2022-08-04 19:04:56 +02:00
Martin Medek
543582e054 DBZ-5165 fixed deployment.yml formatting 2022-08-04 19:04:56 +02:00
Martin Medek
89812a614f DBZ-5165 moved the incremental build into testsuite script, fixed connect image version in deployment 2022-08-04 19:04:56 +02:00
Martin Medek
385d49fc9c DBZ-5165 changed git pull preparations in Dockerfile 2022-08-04 19:04:56 +02:00
Martin Medek
3d313a621f DBZ-5165 changed logging for isRunningFromOcp conditions 2022-08-04 19:04:56 +02:00
Martin Medek
5614712628 DBZ-5165 Refactored mysql test to not create mysql loadBalancer when running inside openshift, refactored the docker testsuite for running inside openshift. Moved project checkout and compilation to Dockerfile, so that it's not always done from scratch 2022-08-04 19:04:56 +02:00
harveyyue
2481172309 DBZ-5471 Missing the regex properties validation before start connector of DefaultRegexTopicNamingStrategy 2022-08-04 10:49:29 -04:00
jcechace
ea7219a2c8 DBZ-5465 Ability to detect AMQ Streams operator installed via OLM 2022-08-03 07:44:18 +02:00
Chris Cranford
90a4c0059a
DBZ-5434 Fix test failure 2022-08-01 14:36:09 -04:00
harveyyue
7c93c597f7 DBZ-5434 Translation from mongodb document to kafka connect schema fails when nested arrays contain no elements 2022-08-01 10:07:28 -04:00
Chris Cranford
0ee30e2115
DBZ-4727 Adjust chunk size for pause test 2022-08-01 10:04:42 -04:00
Jiri Novotny
a330b7da66 DBZ-5466 Change default base image of node snapshot preparation job 2022-08-01 15:50:03 +02:00
Chris Cranford
41a9324044 DBZ-5441 Gracefully skip Oracle non-relational tables 2022-07-29 12:56:27 -04:00
Vojtech Juranek
d39e98fc9f DBZ-4727 Document pause and resume incremental snapshot 2022-07-29 12:41:58 -04:00
Vojtech Juranek
2c505c08f5 DBZ-4727 Add Kafka based pause/resume signals
Add pause and resume snapshot signals. See previous commit for more
details. These signal do the same, but are used by read-only MySQL
where sending signals is done by sending Kafka messages.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
3968b6b75f DBZ-4727 Intorduce KafkaSignal interface
In the follow-up commit signal for pause and resume snapshot will be
added. Intorduce `KafkaSignal` interface which would server as general
kafka-based signal.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
c7d8353d7b DBZ-4727 Add snapshot paused JXM metrics
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
  or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
  snapshot was paused. The time adds up - if the snapshot was paused
  e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
  these two paused times.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
ed504173e1 DBZ-4727 Implement incremental snaphost pause/resume
Intorduce two new signals:
* `pause-snapshot` - pauses running incremental snapshot
* `resume-snapshot` - resumes paused incremental snapshot

If the incremental snapshot is running and pause signal is sent,
currently processed chunk is finished and then no other chunk is read.
Snapshot state is kept in running state. Once snapshot is resume,
chunk is reverted to position which was last sent to the broker and
process of reading chunks and emitting `read` events continues until
all tables are read and sent.

This patch handles only table-based signals. MySQL Kafka-based signals
will be addressed later.
2022-07-29 12:41:58 -04:00
Vojtech Juranek
65bfeeccea DBZ-4727 Make wait for snapshot event generic 2022-07-29 12:41:58 -04:00
Bob Roldan
13c9e517ef DBZ-5458 Remove conditionals surrounding Oracle ad hoc/incremental refs 2022-07-29 08:53:01 -04:00
Bob Roldan
9f0d0000fc DBZ-5422 Update based on review comments 2022-07-28 17:38:41 -04:00
Bob Roldan
2d9cbbc6d8 DBZ-5422 Further edits; add links to connector docs 2022-07-28 17:38:41 -04:00
Bob Roldan
94d9363ff1 DBZ-5422 Language update 2022-07-28 17:38:41 -04:00
Bob Roldan
6de5321c1c DBZ-5422 Update link format and multiple edits 2022-07-28 17:38:41 -04:00
Vojtech Juranek
e70ceeeacd DBZ-5307 Fix adding events from WireMock
In existing test we added all events in the loop which can lead into
adding one event multiple times and therefore failing the tests.
When test run alone, this probably never happen or it was very rare.
Now, when test runs with Debezium server already running, it fails more
often. Fix it by removing already added events from WireMock.
2022-07-28 14:19:13 -04:00
Vojtech Juranek
37d6d3685b DBZ-5307 Update HTTP consumer documentation 2022-07-28 14:19:13 -04:00
Vojtech Juranek
75e5675375 DBZ-5307 Retry upon failure when sending to HTTP sink
If sending record to HTTP consumer fails, retry sending couple of times
and fail eventually. Retrying immdiatelly usually have no effect,
therefore sleep some time between retries.

Intorduce new config option to `HttpChangeConsumer`:
* `retries` - number of retries before exception is throws
* `retry.interval.ms` - amount of time to sleep before next retry
2022-07-28 14:19:13 -04:00
Vojtech Juranek
0e165834bd DBZ-5307 Add method for resetting WireMock
Normally WireMock is reset after each test. However, as we run WireMock
as a container, it's not reset and we have to do so manully. Add
a method which resets WireMock by sending HTTP request.
2022-07-28 14:19:13 -04:00
Vojtech Juranek
72f8388bba DBZ-5307 Refactor HttpChangeConsumer
Refactor HttpChangeConsumer and its test to prepare for further changes
in follow-up patch. Move sending of records into separate method. Also
assert events in the tests in dedicated method.
2022-07-28 14:19:13 -04:00
harveyyue
5f430f1a8e DBZ-5457 Deprecate legacy topic selector for all connectors 2022-07-28 12:58:59 -04:00
Gunnar Morling
a288e2803a [docs] Removing obsolete ZK reference from tutorial 2022-07-27 12:39:20 -04:00
Gunnar Morling
31ade1dda0 [docs] Avoiding reference to Guava types 2022-07-27 12:19:56 -04:00
Martin Medek
2cb482fe17 DBZ-5412 changed mailer to debezium-qe 2022-07-27 13:25:59 +02:00
Martin Medek
9a0c0e05cf DBZ-5412 replaced the old Oracle connector tests job with new one 2022-07-27 13:25:59 +02:00
Martin Medek
0f3b648e64 DBZ-5412 added new job to run oracle connector matrix as real matrix job 2022-07-27 13:25:59 +02:00
ggaborg
2c8d71a36f DBZ-5452 Fix unit test failing on Windows platform 2022-07-26 15:50:28 -04:00
gongzhongqiang
7000f06aac DBZ-5450 Typo in postgresql document. 2022-07-26 15:45:53 -04:00
Debezium Builder
d1ae85d7f0 [release] Development version for testing module deps 2022-07-26 18:35:31 +00:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
Debezium Builder
f0ef85f1dc [release] Stable 2.0.0.Beta1 for testing module deps 2022-07-26 18:06:30 +00:00
Chris Cranford
cf92aa2dcc
[release] Update contributor aliases 2022-07-26 13:58:16 -04:00
Chris Cranford
200776524a [release] Changelog for 2.0.0.Beta1 2022-07-26 13:53:00 -04:00
harveyyue
8d76ba217f DBZ-5333 Caused by: java.io.EOFException: Failed to read next byte from position 2005308603 2022-07-26 11:10:29 -04:00