alexey.rassokhin/tet123
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
7,746 Commits 3 Branches 0 Tags 51 MiB
08119e5368
Commit Graph

1218 Commits

Author SHA1 Message Date
Vojtech Juranek
acc6be6df3 DBZ-5043 Move TOPIC_PREFIX into CommonConnectorConfig 
`TOPIC_PREFIX` is now mandatory to all connectors therefore it make
sense to have it in common config. Beside that, it also makes it more
easy to use it in Debezium UI without any workarounds - if the field is
not member of the given connector config, the field has to be
explicitely added into known fields otherwise is invisible for UI.
 2022-09-13 09:25:54 +02:00
Anisha Mohanty
f41b6dd458 DBZ-5044 Get rid of schema factory instances 2022-09-08 10:35:20 +02:00
Anisha Mohanty
b0dc7e557a DBZ-5044 Update schemaHistoryColumnSchema to add new fields 2022-09-08 10:35:20 +02:00
Anisha Mohanty
ec792527e9 DBZ-5044 Use SchemaFactory for centralizing Postgres schemas 2022-09-08 10:35:20 +02:00
Vojtech Juranek
0823cedf25 DBZ-5043 Replace database.server.name with topic.prefix 2022-09-07 12:57:31 +02:00
Vojtech Juranek
161a7229b6 DBZ-5043 Use topic prefix as the logical name 
With this change the user cannot direcly set connector logical name and
thus in the future it can be remove without breaking user config.
If the turn out that the logical name is useful and user should be able
to configure it, dedicated config option can be added.

Originally it was prposed in the Jira to replace it with connector name,
but it turned out that logical name defaults to `database.server.name`
and is heavily used in the tests and JMX, so it would require another
big refactoring. Thus, use topic prefix for now. Once we know further
direction (remove logical name or add new option), do this refactoring.
 2022-09-07 12:57:31 +02:00
Vadzim Ramanenka
917de37a01 DBZ-5557: Supply partition when comiting offsets with source database 
Keep track of partitions offsets being committed belong to in
`BaseSourceTask`. Supply the partition along with the offset in calls to
`commitOffset`.
 2022-09-05 14:04:23 +02:00
ggaborg
244de2bd96 DBZ-3293 Add signal table automatically to include list 2022-09-01 20:16:47 +02:00
Marek Winkler
1c9ea5166c DBZ-3007: limit 'Streaming requested from LSN' logging 2022-08-31 11:45:47 -04:00
Jiri Pechanec
de53591dba DBZ-5244 Retry all communication exceptions by default 2022-08-31 09:45:09 -04:00
pkgonan
b031770300 DBZ-5544 Support BASE64_URL_SAFE in BinaryHandlingMode 2022-08-24 09:50:23 +02:00
Sergei Morozov
1c99437890 DBZ-5470: Remove the dependency of JdbcConnection on DatabaseSchema 2022-08-19 11:06:59 +02:00
Chris Cranford
37b97970a6 DBZ-5497 Skip truncate operations by default 
* Promote truncate skip default from PostgreSQL to core library
* Adjust behavior for Oracle with truncates skipped by default
 2022-08-19 11:05:29 +02:00
Vojtech Juranek
db446f85c4 DBZ-4329 Don't send fileds with obsolete values 
During incremental snapshot fields `xmin`, `lsn` and `txId` are not
updated and therefore we stream obsolete values from previous records
in `source` struct. Don't include these obsolete value in the `source`
struct during incremental snapshot.
 2022-08-05 10:23:47 -04:00
Vojtech Juranek
c7d8353d7b DBZ-4727 Add snapshot paused JXM metrics 
Add two JXM objects:
* `snapshotPaused` - detemines if the incremental snapshot is paused
  or not
* `snapshotPausedDurationInSeconds` - overall time when the incremental
  snapshot was paused. The time adds up - if the snapshot was paused
  e.g. two times, the `snapshotPausedDurationInSeconds` is the sum of
  these two paused times.
 2022-07-29 12:41:58 -04:00
harveyyue
5f430f1a8e DBZ-5457 Deprecate legacy topic selector for all connectors 2022-07-28 12:58:59 -04:00
Debezium Builder
1ab51ae8d0 [maven-release-plugin] prepare for next development iteration 2022-07-26 18:13:10 +00:00
Debezium Builder
d247a48b79 [maven-release-plugin] prepare release v2.0.0.Beta1 2022-07-26 18:13:09 +00:00
harveyyue
187ccb25db DBZ-5398 The column is referenced as PRIMARY KEY, but a matching column is not defined in table 2022-07-25 10:11:43 -04:00
Vojtech Juranek
3527479cd8 DBZ-5408 Make sure streaming is fully running before dropping connection 2022-07-14 11:10:22 -04:00
Chris Cranford
67dbd189a6 DBZ-5403 Convert timestamps with tz default values to GMT 2022-07-14 08:23:06 +02:00
harveyyue
6ea7abc8fc DBZ-4180 Pluggable topic selector 2022-07-13 14:45:25 +02:00
harveyyue
7af6037078 DBZ-5384 org.postgresql.util.PSQLException: Bad value for type timestamp/date/time: CURRENT_TIMESTAMP 2022-07-12 17:29:15 +02:00
Vojtech Juranek
5573bb51ed DBZ-5048 Rename GH repo with containes 
Besides updating links to renamed GH repo, update also documentation and
replace 'Docker' with 'containers' when possible.
 2022-07-12 13:16:53 +02:00
Chris Cranford
d2a76a7ffc DBZ-5295 Reselect LOB columns on primary key update 2022-07-12 11:59:34 +02:00
Chris Cranford
bb0a640219 DBZ-5340 Add test case 2022-07-08 06:54:18 +02:00
Chris Cranford
c909dd78d2 DBZ-5340 Allow schema prefix on default value function calls 2022-07-08 06:54:18 +02:00
Debezium Builder
6fd1441af3 [maven-release-plugin] prepare for next development iteration 2022-07-01 09:21:49 +00:00
Debezium Builder
99402af634 [maven-release-plugin] prepare release v2.0.0.Alpha3 2022-07-01 09:21:48 +00:00
yangrong688
774d1e4141 [docs] Fix typo for javadoc newSlotInfo 2022-06-30 12:27:06 +02:00
Vojtech Juranek
499aece7eb DBZ-5240 Add partitioned tables into supported table types 
Postgres supports partitioned tables. Debezium requires tables to have
primary key to be able to snapshot them. Primary key support for
partitioned was added in Postgres 11, see [1]. Add partitioned table
type into supported table tables so that Debezium fetches the schema
and can do the snapshot for partitioned tables.

N.B.: On Postgres < 11 is still possible to define primary key
constraint on partitioned sub-tables. In such cases sub-tables are
snapshotted, but parent table is not.

Fix RecordsSnapshotProducerIT#shouldGenerateSnapshotsForPartitionedTables
test after adding support for partitioned tables. Test assumed that
the parent table is not snapshotted and filters out duplicate records
which results into lossing `LAST` snapshot record. Keepting original
test by filtering parent table.

[1] https://www.postgresql.org/docs/11/release-11.html
 2022-06-30 10:46:54 +02:00
Mickael Maison
5ff71aa992 DBZ-5335 Fix formatting issues in the connectors configdefs 2022-06-30 09:35:54 +02:00
harveyyue
20dc353262 DBZ-5170 Mysql Commit Timestamp 2022-06-27 15:50:52 +02:00
kanha gupta
1b17372d2d DBZ-4902 Replace long durations with Duration 2022-06-27 14:29:51 +02:00
Vojtech Juranek
e15f148deb DBZ-5070 Don't reset lsn when lastCommitLsn is null 
Currently we update first `lsn` and then update `lastCommitLsn` in
`PostgresOffsetContext` constructor. However, when `lastCommitLsn`
is for whatever reason `null`, previously updated `lsn` is reset to
`null` as `updateLastCommit()` updates also `lsn`. This can have
unwanted consequences like streaming again records which were already
streamed. To prevent this, update `lsn` in `updateLastCommit()` only
when `lastCommitLsn` is not `null`.
 2022-06-22 13:55:17 +02:00
Jiri Pechanec
4fc4c8c374 DBZ-5031 Stop connector on failure to find resume LSN 2022-06-20 12:00:05 +02:00
Oskar Polak
3e288b8385 DBZ-5031 Additional logging when offset flushing is not allowed 2022-06-20 12:00:05 +02:00
Jiri Pechanec
38cd6907ae DBZ-3942 Load correct last commit LSN from offsets 2022-06-17 14:49:44 +02:00
Jiri Pechanec
04c474bac7 DBZ-5031 Enable LSN flushing only after processed message 2022-06-17 12:39:56 +02:00
Jiri Pechanec
5721dd7ace DBZ-5031 Emit warning if not exact LSN is found 2022-06-17 12:39:56 +02:00
Jun Zhao
29fbdc03bf DBZ-5031 Stop skipping message if last received lsn is higher or equal to the startStreamingLsn in WalPositionLocation 2022-06-17 12:39:56 +02:00
Chris Cranford
3fb5bf1339 DBZ-4451 Introduce SchemaChangeEvent factory methods 2022-06-16 09:19:41 +02:00
Chris Cranford
863abcd081 DBZ-4451 Store previous table id in alter schema change events 
When a table is renamed with the ALTER TABLE statement, the schema
history record not only will reference the current table's unique id
in the "id" metadata, but will also refer to the old table name in
the "previousId" metadata field.
 2022-06-16 09:19:41 +02:00
Chris Cranford
20bb2adea5 DBZ-4834 Fix incremental snapshot of table added to include list 2022-06-16 08:39:16 +02:00
Debezium Builder
6fff3a2876 [maven-release-plugin] prepare for next development iteration 2022-06-09 11:49:26 +00:00
Debezium Builder
b2e2e351df [maven-release-plugin] prepare release v2.0.0.Alpha2 2022-06-09 11:49:26 +00:00
Chris Cranford
660e391851 DBZ-5204 Do not cache connection details in TypeRegistry. 2022-06-08 09:21:04 +02:00
Vojtech Juranek
4a48eb33df DBZ-5045 Remove legacy V1 source info 2022-06-08 09:11:00 +02:00
Vojtech Juranek
7b8cb693ed DBZ-5045 Remove deprecated Postgres TOASTED_VALUE_PLACEHOLDER 2022-06-08 09:11:00 +02:00
Vojtech Juranek
1eacbdc590 DBZ-5045 Remove deprecated Postgres EXPORTED snapshot option 2022-06-08 09:11:00 +02:00
Vojtech Juranek
83dfd9f638 DBZ-5045 Remove deprecated Postgres TRUNCATE_HANDLING_MODE 2022-06-08 09:11:00 +02:00
Vojtech Juranek
b0b3066ed5 DBZ-5045 Remove deprecated COLUMN_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
5031594b75 DBZ-5045 Remove deprecated COLUMN_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
f84893a8b4 DBZ-5045 Remove deprecated SCHEMA_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
261188482f DBZ-5045 Remove deprecated SCHEMA_WHITELIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
6abbcee6a3 DBZ-5045 Remove deprecated TABLE_BLACKLIST 2022-06-08 09:11:00 +02:00
Vojtech Juranek
db08e894fc DBZ-5045 Remove deprecated TABLE_WHITELIST 2022-06-08 09:11:00 +02:00
harveyyue
acb743bd34 DBZ-5206 Remove the deprecated monitored tables metrics 2022-06-07 16:13:29 -04:00
Chris Cranford
9d71296b5c DBZ-4251 Support stopping incremental snapshots 2022-06-02 07:30:07 +02:00
Jiri Pechanec
88b7ae2050 DBZ-5136 Move sourceInfo to common class 2022-06-01 16:56:29 +02:00
Mark Bereznitsky
c64b8af31a DBZ-5136 reintroduce the CommonOffsetContext class and refactor common functionality into it 2022-06-01 16:56:29 +02:00
Mark Bereznitsky
2d72dae649 DBZ-5136 further refining of the algorithm for determining position in snapshot and IT test fixes 2022-06-01 16:56:29 +02:00
Jiri Pechanec
08e8c3d795 DBZ-5136 Move common class to common package 2022-06-01 16:56:29 +02:00
Andrew Walker
d0759461a4 DBZ-5136: Convert RelationalOffsetContext to CommonOffsetContext 2022-06-01 16:56:29 +02:00
Andrew Walker
b6222d42bf DBZ-5136: Add first flag and refactor OffsetContext 2022-06-01 16:56:29 +02:00
Andrew Walker
593c2769e6 DBZ-5136: Add last in table flag 2022-06-01 16:56:29 +02:00
Vojtech Juranek
0757c966d0 DBZ-5042 Add DB name logging support into Postgres connector 2022-05-27 14:20:30 -04:00
Chris Cranford
7d8c9486d6 DBZ-5119 Move Heartbeat creation to ConnectorConfig & descendants 2022-05-25 09:59:59 +02:00
Chris Cranford
7ff7826743 DBZ-5119 Pull heartbeat.action.query support into debezium-core 2022-05-24 12:54:40 +02:00
Chris Cranford
7eee39800f DBZ-5038 Fix test failures 2022-05-23 11:46:15 +02:00
Chris Cranford
2a4c76cabe DBZ-5038 Support default values for PostgreSQL enum types 2022-05-23 11:46:15 +02:00
Jiri Pechanec
a11767d05b DBZ-3921 Remove incorrect comment 2022-05-17 14:56:05 +02:00
troeselereos
836632e5dd DBZ-3921 fixed logging 
DBZ-3921:  update message

Co-authored-by: Chris Cranford <ccranfor@redhat.com>
 2022-05-17 14:56:05 +02:00
roeselert
6b72d22be8 DBZ-3921: fix postgres existing publication is not updated with the new table 2022-05-17 14:56:05 +02:00
Jiri Pechanec
cc90b1cfeb DBZ-5014 Fix delete tests 2022-05-09 08:53:59 +02:00
Cesar Martinez
61a77c2467 DBZ-5014 Remove unused import 2022-05-09 08:53:59 +02:00
Cesar Martinez
c0ac30593c DBZ-5014 Uncomment validations 2022-05-09 08:53:59 +02:00
Cesar Martinez
72e51e02c7 DBZ-5014 Remove invalid line 2022-05-09 08:53:59 +02:00
Cesar Martinez
6994c1db56 DBZ-5014 Test case for delete register with infinity date 2022-05-09 08:53:59 +02:00
Cesar Martinez
840979a454 DBZ-5014 Add AbstractRecordsProducerTest.tableIdFromDeleteStmt(String) 2022-05-09 08:53:59 +02:00
Cesar Martinez
46f6db2d50 DBZ-5014 Add infinity date constants (Date & LocalDate 2022-05-09 08:53:59 +02:00
Cesar Martinez
24173cc548 DBZ-5014 Parse infinity date on delete 2022-05-09 08:53:59 +02:00
Cesar Martinez
b1f7fa1a47 DBZ-5014 Add test case for (+/-)infinity date 2022-05-09 08:53:59 +02:00
Jiri Pechanec
6e092d39f2 DBZ-5074 No separate connection for pgoutput decoder 2022-05-06 16:36:28 +02:00
Jiri Pechanec
88dab4d8d3 DBZ-5074 Track connections by name; close heartbeat connection by default 2022-05-06 16:36:28 +02:00
Mark Allanson
0edb86d861 DBZ-5097: Add tests to cover PostgresErrorHandler behaviour 
The error handler had no unit tests at all, so I thought it might be a good idea
to add some tests to verify the behaviour given there's a bunch of different
ways to determine if something is retryable.
 2022-05-06 09:55:06 -04:00
Mark Allanson
dd815f5226 DBZ-5097: Add two new retryable postgresql error messages 
These two error messages can occur when running inside docker and the
postgres container is either killed (sigtermed) or brought down
using `docker stop`.

In both cases these are retryable as the postgres server can be
restarted shortly thereafter.
 2022-05-06 09:55:06 -04:00
Debezium Builder
901825810a [maven-release-plugin] prepare for next development iteration 2022-04-28 13:51:13 +00:00
Debezium Builder
05dcb45647 [maven-release-plugin] prepare release v2.0.0.Alpha1 2022-04-28 13:51:12 +00:00
Jiri Pechanec
d186ec800f DBZ-703 Remove support for messages without metadata 2022-04-22 09:57:15 +02:00
Jiri Pechanec
d299c9e707 DBZ-4156 Remove wal2json support 2022-04-22 09:57:15 +02:00
Mohammad Yousuf Minhaj Zia
6f706fc087 DBZ-4948 Added connection has been closed PSQLException as retriable 2022-04-07 12:35:27 +02:00
Debezium Builder
4a6de64c25 [maven-release-plugin] prepare for next development iteration 2022-04-05 14:49:52 +00:00
Debezium Builder
683851fd90 [maven-release-plugin] prepare release v1.9.0.Final 2022-04-05 14:49:51 +00:00
Jiri Pechanec
1355067ef0 DBZ-4814 Use schema not catalog in table id 2022-03-31 15:26:12 +02:00
zalmane
1f64e8f8da DBZ-4814 exclude spatial_ref_sys 2022-03-30 20:08:38 +02:00
Debezium Builder
ab7fe3b96a [maven-release-plugin] prepare for next development iteration 2022-03-25 16:45:06 +00:00
Debezium Builder
d19fc234ee [maven-release-plugin] prepare release v1.9.0.CR1 2022-03-25 16:45:06 +00:00
Sergei Morozov
b258674782 DBZ-3535: Introduce schema.name.adjustment.mode 2022-03-24 09:37:05 +01:00
Sergei Morozov
3a03ff0186 DBZ-3535: Remove unused EventDispatcher constructor parameter 2022-03-24 09:37:05 +01:00