Commit Graph

307 Commits

Author SHA1 Message Date
ggaborg
4a70dccd7d DBZ-4720 Unification and validation for Redis sink, offset storage and schema history configuration 2022-11-22 14:46:55 +01:00
ggaborg
a6b2f0db3c DBZ-4720 Validate Debezium Server configuration properties 2022-11-22 14:46:55 +01:00
Debezium Builder
e311dd1a67 [maven-release-plugin] prepare for next development iteration 2022-11-10 13:29:06 +00:00
Debezium Builder
e3bf53440a [maven-release-plugin] prepare release v2.1.0.Alpha1 2022-11-10 13:29:05 +00:00
ggaborg
fdeb497c42 DBZ-5782 constants 2022-11-09 07:16:40 +01:00
ggaborg
1d0b51d2b4 DBZ-5782 rename to memory.threshold.percentage 2022-11-09 07:16:40 +01:00
ggaborg
85bfbc1d67 DBZ-5782 Redis sink back-pressure mechanism when Redis memory is almost full 2022-11-09 07:16:40 +01:00
ggaborg
fcb15dc722 DBZ-5752 Redis Sink wait for Redis Replica writes 2022-11-03 15:43:42 +01:00
jcechace
5b30e1bde8 DBZ-5779 Migrated debezium-server to AssertJ 2022-11-02 08:50:26 +01:00
Chris Cranford
eaab19539f DBZ-5774 Align debezium-server-pravega io.netty dependency with Quarkus 2022-10-31 14:11:41 +01:00
Jiri Pechanec
27441b87da DBZ-5749 Convert Redis store into standard store module 2022-10-24 09:14:50 -04:00
Jiri Pechanec
a51df5e55c DBZ-5737 Upgrade Pulsar to 2.10.1 2022-10-17 13:44:26 +02:00
Debezium Builder
4d35fb8011 [maven-release-plugin] prepare for next development iteration 2022-10-14 11:26:32 +00:00
Debezium Builder
7bd1551c99 [maven-release-plugin] prepare release v2.0.0.Final 2022-10-14 11:26:32 +00:00
Jiri Pechanec
47bdbbcaad DBZ-5711 Prevent Quarkus from overriding driver version 2022-10-13 08:48:52 +02:00
Debezium Builder
884254aba1 [maven-release-plugin] prepare for next development iteration 2022-10-07 09:27:52 +00:00
Debezium Builder
a0f3263fa6 [maven-release-plugin] prepare release v2.0.0.CR1 2022-10-07 09:27:51 +00:00
Jiri Pechanec
99dff302f1 DBZ-5612 Support for formatted messages 2022-10-04 13:45:43 +02:00
rajdangwal
58ab632ea3 DBZ-5612 log records only at trace level 2022-10-04 13:45:43 +02:00
rajdangwal
c03bcf8606 DBZ-5612 Remove records from being logged at all levels 2022-10-04 13:45:43 +02:00
Debezium Builder
218d414552 [maven-release-plugin] prepare for next development iteration 2022-09-16 06:36:40 +00:00
Debezium Builder
567f57c47b [maven-release-plugin] prepare release v2.0.0.Beta2 2022-09-16 06:36:39 +00:00
Vojtech Juranek
402df78381 DBZ-5043 Rename schema.history to schema.history.internal 2022-09-07 12:57:31 +02:00
Vojtech Juranek
0235d4aad5 DBZ-5582 Fix RedisDatabaseHistoryIT tests
`RedisSchemaHistoryIT` tests are not reliable as
`testDatabaseHistoryIsSaved` makes assumption about the number of
entries in the database history stream, while
`testRedisConnectionRetry` deletes whole stream at the beginning of
the test. If `testDatabaseHistoryIsSaved` is executed after
`testRedisConnectionRetry`, `testDatabaseHistoryIsSaved` starts to fail.

Don't delete the stream in `testRedisConnectionRetry` and introduce
constant with initial number of entries in the schema history stream.
2022-09-07 12:57:31 +02:00
Vojtech Juranek
0494a16a79 DBZ-5043 Rename DatabaseHistory to SchemaHistory
Rename also all related classes and varibales.
2022-09-07 12:57:31 +02:00
Vojtech Juranek
0823cedf25 DBZ-5043 Replace database.server.name with topic.prefix 2022-09-07 12:57:31 +02:00
Vojtech Juranek
25aa6c4acc DBZ-5043 Change namespace to database.history to schema.history 2022-09-07 12:57:31 +02:00
ggaborg
196d24f2be DBZ-4441 tests for default redis message format 'compact' 2022-09-06 06:21:58 +02:00
ggaborg
f97e30bfdb DBZ-4441 make 'compact' the default redis message format for backward compatibility 2022-09-06 06:21:58 +02:00
ggaborg
e500cad38f DBZ-4441 placeholders for messages 2022-09-06 06:21:58 +02:00
ggaborg
9243fae507 DBZ-4441 Redis Sink - Change the format of the message sent to the stream 2022-09-06 06:21:58 +02:00
ggaborg
abe2a5e910 DBZ-5122 Include Oracle Debezium Connector in Debezium Server distribution 2022-08-25 09:17:05 +02:00
ggaborg
fcd2664d2e DBZ-5513 constant for offset.storage prefix 2022-08-17 15:44:39 +02:00
ggaborg
6c3955aa01 DBZ-5513 Simplify passing of SINK config properties to OffsetBackingStore 2022-08-17 15:44:39 +02:00
ggaborg
727e8819e9 DBZ-5506 duplicate call to config.validateAndRecord() in RedisDatabaseHistory 2022-08-16 08:31:48 -04:00
ggaborg
3489217e0d DBZ-5035 updated test 2022-08-15 13:29:12 +02:00
ggaborg
469d7b9aec DBZ-5035 Redis Sink config properties are not passed to DB history 2022-08-15 13:29:12 +02:00
Vojtech Juranek
43787b8a21 DBZ-5116 Provide more clear message when cannot load config
Example output after the failure:

    $ ./run.sh
    Failed to load mandatory config value 'debezium.sink.type'. Please check you have correct Debezium server config in /home/vjuranek/tmp/debezium-server/conf/application.properties or required properties are defined via system or environment variables.
    {"timestamp":"2022-08-08T13:37:35.848+02:00","sequence":155,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus.runtime.Application","level":"ERROR","message":"Failed to start application (with profile prod)","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"fedora","processName":"io.debezium.server.Main","processId":1048188,"exception":{"refId":1,"exceptionType":"java.util.NoSuchElementException","message":"SRCFG00014: The config property debezium.sink.type is required but it could not be found in any config source","frames":[{"class":"io.smallrye.config.SmallRyeConfig","method":"convertValue","line":282},{"class":"io.smallrye.config.SmallRyeConfig","method":"getValue","line":239},{"class":"io.smallrye.config.SmallRyeConfig","method":"getValue","line":167},{"class":"io.debezium.server.DebeziumServer","method":"start","line":104},{"class":"io.debezium.server.DebeziumServer_Bean","method":"create"},{"class":"io.debezium.server.DebeziumServer_Bean","method":"create"},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"createInstanceHandle","line":111},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":35},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":32},{"class":"io.quarkus.arc.impl.LazyValue","method":"get","line":26},{"class":"io.quarkus.arc.impl.ComputingCache","method":"computeIfAbsent","line":69},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"get","line":32},{"class":"io.quarkus.arc.impl.ClientProxies","method":"getApplicationScopedDelegate","line":19},{"class":"io.debezium.server.DebeziumServer_ClientProxy","method":"arc$delegate"},{"class":"io.debezium.server.DebeziumServer_ClientProxy","method":"arc_contextualInstance"},{"class":"io.debezium.server.DebeziumServer_Observer_Synthetic_d70cd75bf32ab6598217b9a64a8473d65e248c05","method":"notify"},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notifyObservers","line":323},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notify","line":305},{"class":"io.quarkus.arc.impl.EventImpl","method":"fire","line":73},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"fireLifecycleEvent","line":130},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"handleLifecycleEvents","line":99},{"class":"io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy_0"},{"class":"io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy"},{"class":"io.quarkus.runner.ApplicationImpl","method":"doStart"},{"class":"io.quarkus.runtime.Application","method":"start","line":101},{"class":"io.quarkus.runtime.ApplicationLifecycleManager","method":"run","line":103},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":67},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":41},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":120},{"class":"io.debezium.server.Main","method":"main","line":15}]}}
2022-08-08 10:24:56 -04:00
Vojtech Juranek
e70ceeeacd DBZ-5307 Fix adding events from WireMock
In existing test we added all events in the loop which can lead into
adding one event multiple times and therefore failing the tests.
When test run alone, this probably never happen or it was very rare.
Now, when test runs with Debezium server already running, it fails more
often. Fix it by removing already added events from WireMock.
2022-07-28 14:19:13 -04:00
Vojtech Juranek
75e5675375 DBZ-5307 Retry upon failure when sending to HTTP sink
If sending record to HTTP consumer fails, retry sending couple of times
and fail eventually. Retrying immdiatelly usually have no effect,
therefore sleep some time between retries.

Intorduce new config option to `HttpChangeConsumer`:
* `retries` - number of retries before exception is throws
* `retry.interval.ms` - amount of time to sleep before next retry
2022-07-28 14:19:13 -04:00
Vojtech Juranek
0e165834bd DBZ-5307 Add method for resetting WireMock
Normally WireMock is reset after each test. However, as we run WireMock
as a container, it's not reset and we have to do so manully. Add
a method which resets WireMock by sending HTTP request.
2022-07-28 14:19:13 -04:00
Vojtech Juranek
72f8388bba DBZ-5307 Refactor HttpChangeConsumer
Refactor HttpChangeConsumer and its test to prepare for further changes
in follow-up patch. Move sending of records into separate method. Also
assert events in the tests in dedicated method.
2022-07-28 14:19:13 -04:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
Jiri Pechanec
ad7e65fbac DBZ-5235 Use JMX to get queue size 2022-07-12 16:40:48 +02:00
Ismail Simsek
d84559f147 DBZ-5235 read Debezium metrics from consumer 2022-07-12 16:40:48 +02:00
Debezium Builder
6fd1441af3 [maven-release-plugin] prepare for next development iteration 2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634 [maven-release-plugin] prepare release v2.0.0.Alpha3 2022-07-01 09:21:48 +00:00
kanha gupta
1b17372d2d DBZ-4902 Replace long durations with Duration 2022-06-27 14:29:51 +02:00
Anisha Mohanty
27d17fd77e DBZ-5297 Upgrade azure-messaging-eventhubs to 5.12.1 version 2022-06-22 12:04:00 -04:00