This single message transformation (SMT) is under active development right now, so the emitted message structure or other details may still change as development progresses.
Please see below for a descriptions of known limitations of this transformation.
====
[NOTE]
====
This SMT is supported only for the MongoDB connector.
See xref:{link-event-flattening}[Extracting source record `after` state from {prodname} change events] for the relational database equivalent to this SMT.
The connector might emit many types of event messages (for example, heartbeat messages, tombstone messages, or metadata messages about transactions).
To apply the transformation to a subset of events, you can define xref:options-for-applying-the-transformation-selectively[an SMT predicate statement that selectively applies the transformation] to specific events only.
This option allows you to process arbitrary arrays but the consumer need to know how to properly handle them.
_Note: The underscore in index names is present because Avro encoding requires field names not to start with digit._
=== Nested structure flattening
When a MongoDB document contains a nested document (structure) it is faithfully encoded as a nested structure field.
If the sink connector does support only flat structure it is possible to flatten the internal structure into a flat one with a consistent field naming.
To enable this feature the option `flatten.struct` must be set to `true`.
The resulting flat document will consist of fields whose names are created by joining the name of the parent field and the name of the fields in the nested document.
Those elements are separated with string defined by an option `struct.delimiter` by default set to the _underscore_.
Let's suppose an example source MongoDB document with a field with a nested document
MongoDB allows `$unset` operations that remove a certain field from a document. Because the collections are schemaless, it becomes hard to inform consumers/sinkers about the field that is now missing. The approach that {prodname} uses is to set the field being removed to a null value.
When a message is flattened the final result does not show whether it was an insert, update or first read. (Deletions can be detected via tombstones or rewrites, see xref:{link-mongodb-event-flattening}#mongodb-extract-new-record-state-configuration-options[Configuration options].)
This ability to add metadata to the event record makes it possible to include content such as the name of the collection associated with the change event, or such connector-specific fields as the replica set name.
For example, you might specify the following configuration to add a replica set name (`rs`) and the collection name for a change event to the final flattened event record:
In addition to the change event messages that a {prodname} connector emits when a database change occurs, the connector also emits other types of messages, including heartbeat messages, and metadata messages about schema changes and transactions.
Because the structure of these other messages differs from the structure of the change event messages that the SMT is designed to process, it's best to configure the connector to selectively apply the SMT, so that it processes only the intended data change messages.
For more information about how to apply the SMT selectively, see xref:{link-smt-predicates}#applying-transformations-selectively[Configure an SMT predicate for the transformation].
|Delimiter to concat between field names from the input record when generating field names for the output record. Only applies when `flatten.struct` is set to `true`
|The SMT can `drop`, `rewrite` or pass delete records (`none`). The `rewrite` mode will add a `__deleted` field set to `true` or `false` depending on the represented operation.
* Feeding data changes from a schemaless store such as MongoDB to strictly schema-based datastores such as a relational database can by definition work within certain limits only.
Specifically, all fields of documents within one collection with the same name must be of the same type. Otherwise, no consistent column definition can be derived in the target database.
* Arrays will be restored in the emitted Kafka Connect record correctly, but they are not supported by sink connector just expecting a "flat" message structure.