This SMT is for use with the {prodname} MongoDB connector only.
For information about using the outbox event router SMT for relational databases, see {link-prefix}:{link-outbox-event-router}#outbox-event-router[Outbox event router].
The outbox pattern is a way to safely and reliably exchange data between multiple (micro) services. An outbox pattern implementation avoids inconsistencies between a service's internal state (as typically persisted in its database) and state in events consumed by services that need the same data.
To implement the outbox pattern in a {prodname} application, configure a {prodname} connector to:
* Capture changes in an outbox collection
* Apply the {prodname} MongoDB outbox event router single message transformation (SMT)
A {prodname} connector that is configured to apply the MongoDB outbox SMT should capture changes that occur in an outbox collection only.
For more information, see xref:mongodb-outbox-options-for-applying-the-transformation-selectively[Options for applying the transformation selectively].
To use this SMT, operations on the actual business collection(s) and the insert into the outbox collection must be done as part of a multi-document transaction, which have been being supported since MongoDB 4.0, to prevent potential data inconsistencies between business collection(s) and outbox collection.
For future update, to enable updating existing data and inserting outbox event in an ACID transaction without multi-document transactions, we have planned to support additional configurations for storing outbox events in a form of a sub-document of the existing collection, rather than an independent outbox collection.
For more information about the outbox pattern, see link:https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/[Reliable Microservices Data Exchange With the Outbox Pattern].
A {prodname} connector that is configured to apply the MongoDB outbox event router SMT generates the preceding message by transforming a raw {prodname} change event message as in the following example:
This example of a {prodname} outbox message is based on the xref:mongodb-outbox-event-router-configuration-options[default outbox event router configuration], which assumes an outbox collection structure and event routing based on aggregates.
To customize behavior, the outbox event router SMT provides numerous xref:mongodb-outbox-event-router-configuration-options[configuration options].
To obtain the unique ID of the event from a different outbox collection field, set the xref:mongodb-outbox-event-router-property-collection-field-event-id[`collection.field.event.id`] SMT option in the connector configuration.
|Contains a value that the SMT appends to the name of the topic to which the connector emits an outbox message.
The default behavior is that this value replaces the default `pass:[${routedByValue}]` variable in the xref:mongodb-outbox-event-router-property-route-topic-replacement[`route.topic.replacement`] SMT option. +
For example, in a default configuration, the xref:mongodb-outbox-event-router-property-route-by-field[`route.by.field`] SMT option is set to `aggregatetype` and the xref:mongodb-outbox-event-router-property-route-topic-replacement[`route.topic.replacement`] SMT option is set to `outbox.event.pass:[${routedByValue}]`.
Suppose that your application adds two documents to the outbox collection. In the first document, the value in the `aggregatetype` field is `customers`.
In the second document, the value in the `aggregatetype` field is `orders`.
The connector emits the first document to the `outbox.event.customers` topic.
The connector emits the second document to the `outbox.event.orders` topic. +
To obtain this value from a different outbox collection field, set the xref:mongodb-outbox-event-router-property-route-by-field[`route.by.field`] SMT option in the connector configuration.
To obtain the event key from a different outbox collection field, set the xref:mongodb-outbox-event-router-property-collection-field-event-key[`collection.field.event.key`] SMT option in the connector configuration.
By default, the Kafka message value is solely comprised of the `payload` value.
However, if the outbox event is configured to include additional fields, the Kafka message value contains an envelope encapsulating both payload and the additional fields, and each field is represented separately.
To obtain the event payload from a different outbox collection field, set the xref:mongodb-outbox-event-router-property-collection-field-event-payload[`collection.field.event.payload`] SMT option in the connector configuration.
|Any additional fields from the outbox collection can be xref:mongodb-outbox-emitting-messages-with-additional-fields[added to outbox events] either within the payload section or as a message header. +
To configure a {prodname} connector to support the outbox pattern, configure the `outbox.EventRouter` SMT. The following example shows the basic configuration for the SMT in a `.properties` file:
== Options for applying the transformation selectively
In addition to the change event messages that a {prodname} connector emits when a database change occurs, the connector also emits other types of messages, including heartbeat messages, and metadata messages about schema changes and transactions.
Because the structure of these other messages differs from the structure of the change event messages that the SMT is designed to process, it's best to configure the connector to selectively apply the SMT, so that it processes only the intended data change messages.
You can use one of the following methods to configure the connector to apply the SMT selectively:
The MongoDB outbox event router SMT supports arbitrary payload formats. The `payload` field value in an outbox collection is passed on transparently. An alternative to working with JSON is to use Avro.
This can be beneficial for message format governance and for ensuring that outbox event schemas evolve in a backwards-compatible way.
How a source application produces Avro formatted content for outbox message payloads is out of the scope of this documentation.
One possibility is to leverage the `KafkaAvroSerializer` class to serialize `GenericRecord` instances.
To ensure that the Kafka message value is the exact Avro binary data,
apply the following configuration to the connector:
By default, the `payload` field value (the Avro data) is the only message value.
Configuration of `ByteBufferConverter` as the value converter propagates the `payload` field value as-is into the Kafka message value.
The {prodname} connectors may be configured to emit heartbeat, transaction metadata, or schema change events (support varies by connector).
These events cannot be serialized by the `ByteBufferConverter` so additional configuration must be provided so the converter knows how to serialize these events.
As an example, the following configuration illustrates using the Apache Kafka `JsonConverter` with no schemas:
The delegate `Converter` implementation is specified by the `delegate.converter.type` option.
If any extra configuration options are needed by the converter, they can also be specified, such as the disablement of schemas shown above using `schemas.enable=false`.
Your outbox collection might contain fields whose values you want to add to the emitted outbox messages. For example, consider an outbox collection that has a value of `purchase-order` in the `aggregatetype` field and another field, `eventType`, whose possible values are `order-created` and `order-shipped`.
To emit the `eventType` field value in the outbox message header, configure the SMT like this:
By default, the `payload` of the {prodname} outbox message is represented as a string.
When the original source of the string is in JSON format, the resulting Kafka message uses escape sequences to represent the string, as shown in the following example:
To enable string conversion in the transformation, set the value of `collection.expand.json.payload` to `true` and use the `StringConverter` as shown in the following example:
All changes in an outbox collection are expected to be an insert or delete operation. That is, an outbox collection functions as a queue; updates to documents in an outbox collection are not allowed.
The SMT automatically filters out delete operations (for removing proceeded outbox events) on an outbox collection.
|Specifies the outbox collection field that contains the event key. When this field contains a value, the SMT uses that value as the key in the emitted outbox message. This is important for maintaining correct order in Kafka partitions.
|By default, the timestamp in the emitted outbox message is the {prodname} event timestamp. To use a different timestamp in outbox messages, set this option to an outbox collection field that contains the timestamp that you want to be in emitted outbox messages.
a|Specifies whether the JSON expansion of a String payload should be done. If no content found or in case of parsing error, the content is kept "as is". +
a|Specifies one or more outbox collection fields that you want to add to outbox message headers or envelopes. Specify a comma-separated list of pairs. In each pair, specify the name of a field and whether you want the value to be in the header or the envelope. Separate the values in the pair with a colon, for example:
`id:header,my-field:envelope`
To specify an alias for the field, specify a trio with the alias as the third value, for example:
`id:header,my-field:envelope:my-alias`
The second value is the placement and it must always be `header` or `envelope`.
|When set, this value is used as the schema version as described in the link:https://kafka.apache.org/20/javadoc/org/apache/kafka/connect/data/ConnectSchema.html#version--[Kafka Connect Schema] Javadoc.
|Specifies a regular expression that the outbox SMT applies in the RegexRouter to outbox collection documents. This regular expression is part of the setting of the xref:mongodb-outbox-event-router-property-route-topic-replacement[`route.topic.replacement`] SMT option. +
+
The default behavior is that the SMT replaces the default `pass:[${routedByValue}]` variable in the setting of the `route.topic.replacement` SMT option with the setting of the xref:mongodb-outbox-event-router-property-route-by-field[`route.by.field`] outbox SMT option.
a|Specifies the name of the topic to which the connector emits outbox messages.
The default topic name is `outbox.event.` followed by the `aggregatetype` field value in the outbox collection document. For example, if the `aggregatetype` value is `customers`, the topic name is `outbox.event.customers`. +
+
To change the topic name, you can: +
* Set the xref:mongodb-outbox-event-router-property-route-by-field[`route.by.field`] option to a different field.