1d241283b7
Co-authored-by: roldanbob <broldan@redhat.com>
142 lines
5.0 KiB
Plaintext
142 lines
5.0 KiB
Plaintext
:page-aliases: configuration/compute-partition.adoc
|
|
// Category: debezium-using
|
|
// Type: assembly
|
|
// ModuleID: routing-records-to-partitions-based-on-column-values
|
|
// Title: Routing records to partitions based on column values
|
|
[id="compute-partition"]
|
|
= [DEPRECATED] Compute Partition
|
|
|
|
:toc:
|
|
:toc-placement: macro
|
|
:linkattrs:
|
|
:icons: font
|
|
:source-highlighter: highlight.js
|
|
|
|
toc::[]
|
|
|
|
[IMPORTANT]
|
|
====
|
|
This page describes an SMT that will be removed in future releases.
|
|
Please use the new SMT xref:transformations/partition-routing.adoc[Partition Routing].
|
|
====
|
|
|
|
By default, when {prodname} detects a change in a data collection, it emits a change event to an Apache Kafka topic with a single partition.
|
|
As described in {link-prefix}:{link-topic-auto-creation}#customizing-debezium-automatically-created-topics[Customization of Kafka Connect automatic topic creation], you can customize the default configuration to route events to multiple partitions, based on a hash of the primary key.
|
|
|
|
However, in some cases, you might want {prodname} to route events to a specific partition.
|
|
The compute partition SMT enables you to route events to destination partitions based on a specified column name value.
|
|
{prodname} uses the hash of the specified value to determine the destination partition.
|
|
|
|
// Type: concept
|
|
// Title: Example: Basic configuration of the {prodname} compute partition SMT
|
|
// ModuleID: basic-configuration-of-the-debezium-compute-partition-smt
|
|
[[example-basic-compute-partition-configuration-example]]
|
|
== Example: Basic configuration
|
|
|
|
You configure the compute partition transformation in the {prodname} connector's Kafka Connect configuration.
|
|
The configuration specifies the following parameters:
|
|
|
|
* The data collection column to use to calculate the destination partition.
|
|
* The maximum number of partitions permitted for the data collection.
|
|
|
|
The SMT only processes events that originate in specified data collections.
|
|
Events from other data collections are ignored.
|
|
|
|
To configure a {prodname} connector to route events to a specific partition, configure the `ComputePartition` SMT in the Kafka Connect configuration for the {prodname} connector.
|
|
|
|
For example, you might add the following configuration in your connector configuration.
|
|
|
|
[source]
|
|
----
|
|
...
|
|
topic.creation.default.partitions=2
|
|
topic.creation.default.replication.factor=1
|
|
...
|
|
transforms=ComputePartition
|
|
transforms.ComputePartition.type=io.debezium.transforms.partitions.ComputePartition
|
|
transforms.ComputePartition.partition.data-collections.field.mappings=inventory.products:name,inventory.orders:purchaser
|
|
transforms.ComputePartition.partition.data-collections.partition.num.mappings=inventory.products:2,inventory.orders:2
|
|
...
|
|
----
|
|
|
|
The configuration in the preceding example enables partition computation for the `products` and `orders` data collections.
|
|
The configuration specifies that the SMT uses the `name` column to compute the partition for the `products` data collection.
|
|
The number of partitions is set to `2`.
|
|
The number of partitions that you specify must match the number of partitions that are specified in the Kafka topic configuration.
|
|
|
|
Based on the configuration in the example, for the following `Products` table, the SMT routes change events for all records that have the field name `hammer` to the same partition.
|
|
That is, the items with `id` values `104`, `105`, and `106` are routed to the same partition.
|
|
|
|
.Products table
|
|
[cols="25%a,25%a,25%a,25%a"]
|
|
|===
|
|
|id
|
|
|name
|
|
|description
|
|
|weight
|
|
|
|
|101
|
|
|scooter
|
|
|Small 2-wheel scooter
|
|
| 3.14
|
|
|
|
|102
|
|
|car battery
|
|
|12V car battery
|
|
| 8.1
|
|
|103
|
|
|12-pack drill bits
|
|
|12-pack of drill bits with sizes ranging from #40 to #3
|
|
| 0.8
|
|
|104
|
|
|hammer
|
|
|12oz carpenter's hammer
|
|
| 0.75
|
|
|105
|
|
|hammer
|
|
|14oz carpenter's hammer
|
|
| 0.875
|
|
|106
|
|
|hammer
|
|
|16oz carpenter's hammer
|
|
| 1.0
|
|
|107
|
|
|rocks
|
|
|box of assorted rocks
|
|
| 5.3
|
|
|108
|
|
|jacket
|
|
|water-resistant black windbreaker
|
|
| 0.1
|
|
|109
|
|
|spare tire
|
|
|24-inch spare tire
|
|
| 22.2
|
|
|===
|
|
|
|
// Type: reference
|
|
// ModuleID: options-for-configuring-the-compute-partition-transformation
|
|
// Title: Options for configuring the compute partition transformation
|
|
[[compute-partition-configuration-options]]
|
|
== Configuration options
|
|
|
|
The following table lists the configuration options that you can use with the compute partition SMT.
|
|
|
|
.Partition routing SMT (`ComputePartition`) configuration options
|
|
[cols="30%a,25%a,45%a"]
|
|
|===
|
|
|Property
|
|
|Default
|
|
|Description
|
|
|
|
|[[compute-partition-data-collections-field-mappings]]<<compute-partition-data-collections-field-mappings, `partition.data-collections.field.mappings`>>
|
|
|
|
|
|A comma-separated list of colon-delimited pairs that specify the columns to use for a specific data collection, for example, `inventory.products:name,inventory.orders:purchaser`.
|
|
|
|
|[[compute-partition-data-collections-partition-num-mappings]]<<compute-partition-data-collections-partition-num-mappings, `partition.data-collections.partition.num.mappings`>>
|
|
|
|
|
|A comma-separated list of colon-delimited pairs that specify the number of partitions to use for a specific data collection, for example, `inventory.products:2,inventory.orders:3`.
|
|
|
|
|
|
|
|===
|