This single message transformation (SMT) is under active development right now, so the emitted message structure or other details may still change as development progresses.
Please see below for a descriptions of known limitations of this transformation.
====
[NOTE]
====
This SMT is supported only for the MongoDB connector.
See xref:configuration/event-flattening.adoc[here] for the relational database equivalent to this SMT.
====
[WARNING]
====
0.9.0 Backwards compatible breaking changes
Breaking changes were introduced together with handling deletions features, the previous default behavior was to keep deletion messages, now the new default behavior is to remove them. To change this setting please refer to link:#configuration_options[`delete.handling.mode`] and link:#configuration_options[`drop.tombstones`]
====
The Debezium MongoDB connector generates the data in a form of a complex message structure.
The message consists of two parts:
* operation and metadata
* for inserts, the whole data after the insert has been executed; for updates a patch element describing the altered fields
The `after` and `patch` elements are Strings containing JSON representations of the inserted/altered data.
E.g. the general message structure for a insert event looks like this:
This option allows you to process arbitrary arrays but the consumer need to know how to properly handle them.
_Note: The underscore in index names is present because Avro encoding requires field names not to start with digit._
=== Nested structure flattening
When a MongoDB document contains a nested document (structure) it is faithfully encoded as a nested structure field.
If the sink connector does support only flat structure it is possible to flatten the internal structure into a flat one with a consistent field naming.
To enable this feature the option `flatten.struct` must be set to `true`.
The resulting flat document will consist of fields whose names are created by joining the name of the parent field and the name of the fields in the nested document.
Those elements are separated with string defined by an option `struct.delimiter` by default set to the _underscore_.
Let's suppose an example source MongoDB document with a field with a nested document
This option allows you to convert a hierarchical document into a flat structure suitable for a table-like storage.
=== MongoDB `$unset` handling
MongoDB allows you to make `$unset` operations which allows you to remove a certain field from a Document, and since the collections are schemaless it becomes hard to find a way to tell the consumers/sinkers which a field is now missing, the approach debezium uses is to set the desired to remove field to null value.
Given the operation
[source,json,indent=0]
----
{
"after":null,
"patch":"{\"$unset\" : {\"a\" : true}}"
}
----
The final encoding will look like
[source,json,indent=0]
----
{
"id": 1,
"a": null
}
----
Note that other mongo operations might cause an `$unset` internally, `$rename` is one example.
=== Determine original operation
When a message is flattened the final result won't show whether it was an insert, update or first read (Deletions can be detected via tombstones or rewrites, see link:#configuration_options[Configuration options]).
To solve this problem Debezium offers an option to propagate the original operation via a header added to the message.
To enable this feature the option `operation.header` must be set to `true`.
|The SMT converts MongoDB arrays into arrays as defined by Apache Connect (or Apache Avro) schema.
|`flatten.struct`
|`false`
|The SMT flattens structs by concatenating the fields into plain properties, using a configurable delimiter.
|`flatten.struct.delimiter`
|`_`
|Delimiter to concat between field names from the input record when generating field names for the output record. Only applies when `flatten.struct` is set to `true`
|`operation.header`
|`false`
|The SMT adds the xref:connectors/mongodb.adoc#change-events-value[event operation] as a message header.
|`drop.tombstones`
|`true`
|The SMT removes the tombstone generated by Debezium from the stream.
|`delete.handling.mode`
|`drop`
|The SMT can `drop`, `rewrite` or pass delete records (`none`). The `rewrite` mode will add a `__deleted` field set to `true` or `false` depending on the represented operation.
* Feeding data changes from a schemaless store such as MongoDB to strictly schema-based datastores such as a relational database can by definition work within certain limits only.
Specifically, all fields of documents within one collection with the same name must be of the same type. Otherwise, no consistent column definition can be derived in the target database.
* Arrays will be restored in the emitted Kafka Connect record correctly, but they are not supported by sink connector just expecting a "flat" message structure.