2023-03-29 16:20:50 +02:00
:page-aliases: configuration/compute-partition.adoc
// Category: debezium-using
// Type: assembly
// ModuleID: routing-records-to-partitions-based-on-column-values
// Title: Routing records to partitions based on column values
[id="compute-partition"]
= [DEPRECATED] Compute Partition
:toc:
:toc-placement: macro
:linkattrs:
:icons: font
:source-highlighter: highlight.js
toc::[]
[IMPORTANT]
====
This page describes an SMT that will be removed in future releases.
Please use the new SMT xref:transformations/partition-routing.adoc[Partition Routing].
====
2023-04-05 09:47:41 +02:00
By default, when {prodname} detects a change in a data collection, it emits a change event to an Apache Kafka topic with a single partition.
2023-03-29 16:20:50 +02:00
As described in {link-prefix}:{link-topic-auto-creation}#customizing-debezium-automatically-created-topics[Customization of Kafka Connect automatic topic creation], you can customize the default configuration to route events to multiple partitions, based on a hash of the primary key.
2023-04-05 09:47:41 +02:00
However, in some cases, you might want {prodname} to route events to a specific partition.
The compute partition SMT enables you to route events to destination partitions based on a specified column name value.
{prodname} uses the hash of the specified value to determine the destination partition.
2023-03-29 16:20:50 +02:00
// Type: concept
// Title: Example: Basic configuration of the {prodname} compute partition SMT
// ModuleID: basic-configuration-of-the-debezium-compute-partition-smt
[[example-basic-compute-partition-configuration-example]]
== Example: Basic configuration
You configure the compute partition transformation in the {prodname} connector's Kafka Connect configuration.
The configuration specifies the following parameters:
* The data collection column to use to calculate the destination partition.
* The maximum number of partitions permitted for the data collection.
2023-04-05 09:47:41 +02:00
The SMT only processes events that originate in specified data collections.
Events from other data collections are ignored.
2023-03-29 16:20:50 +02:00
To configure a {prodname} connector to route events to a specific partition, configure the `ComputePartition` SMT in the Kafka Connect configuration for the {prodname} connector.
For example, you might add the following configuration in your connector configuration.
[source]
----
...
topic.creation.default.partitions=2
topic.creation.default.replication.factor=1
...
transforms=ComputePartition
transforms.ComputePartition.type=io.debezium.transforms.partitions.ComputePartition
transforms.ComputePartition.partition.data-collections.field.mappings=inventory.products:name,inventory.orders:purchaser
transforms.ComputePartition.partition.data-collections.partition.num.mappings=inventory.products:2,inventory.orders:2
...
----
2023-04-05 09:47:41 +02:00
The configuration in the preceding example enables partition computation for the `products` and `orders` data collections.
The configuration specifies that the SMT uses the `name` column to compute the partition for the `products` data collection.
The number of partitions is set to `2`.
The number of partitions that you specify must match the number of partitions that are specified in the Kafka topic configuration.
2023-03-29 16:20:50 +02:00
2023-04-05 09:47:41 +02:00
Based on the configuration in the example, for the following `Products` table, the SMT routes change events for all records that have the field name `hammer` to the same partition.
That is, the items with `id` values `104`, `105`, and `106` are routed to the same partition.
2023-03-29 16:20:50 +02:00
.Products table
[cols="25%a,25%a,25%a,25%a"]
|===
|id
|name
|description
|weight
|101
|scooter
|Small 2-wheel scooter
| 3.14
|102
|car battery
|12V car battery
| 8.1
|103
|12-pack drill bits
|12-pack of drill bits with sizes ranging from #40 to #3
| 0.8
|104
|hammer
|12oz carpenter's hammer
| 0.75
|105
|hammer
|14oz carpenter's hammer
| 0.875
|106
|hammer
|16oz carpenter's hammer
| 1.0
|107
|rocks
|box of assorted rocks
| 5.3
|108
|jacket
2023-04-05 09:47:41 +02:00
|water-resistant black windbreaker
2023-03-29 16:20:50 +02:00
| 0.1
|109
|spare tire
2023-04-05 09:47:41 +02:00
|24-inch spare tire
2023-03-29 16:20:50 +02:00
| 22.2
|===
// Type: reference
// ModuleID: options-for-configuring-the-compute-partition-transformation
// Title: Options for configuring the compute partition transformation
[[compute-partition-configuration-options]]
== Configuration options
The following table lists the configuration options that you can use with the compute partition SMT.
.Partition routing SMT (`ComputePartition`) configuration options
[cols="30%a,25%a,45%a"]
|===
|Property
|Default
|Description
|[[compute-partition-data-collections-field-mappings]]<<compute-partition-data-collections-field-mappings, `partition.data-collections.field.mappings`>>
|
2023-04-05 09:47:41 +02:00
|A comma-separated list of colon-delimited pairs that specify the columns to use for a specific data collection, for example, `inventory.products:name,inventory.orders:purchaser`.
2023-03-29 16:20:50 +02:00
|[[compute-partition-data-collections-partition-num-mappings]]<<compute-partition-data-collections-partition-num-mappings, `partition.data-collections.partition.num.mappings`>>
|
2023-04-05 09:47:41 +02:00
|A comma-separated list of colon-delimited pairs that specify the number of partitions to use for a specific data collection, for example, `inventory.products:2,inventory.orders:3`.
2023-03-29 16:20:50 +02:00
|
|===