{prodname} connectors work with the Kafka Connect framework to capture changes in databases and generate change event records. The Kafka Connect workers then apply any configured transformations to each of the messages generated by the connector, serialize each message key and value into a binary form by using the configured link:https://kafka.apache.org/documentation/#connect_running[_converters_], and write each message into the correct Kafka topic.
Kafka Connect comes with a _JSON converter_ that serializes message keys and values into JSON documents. You can configure the JSON converter to include or exclude the message schema by specifying the `key.converter.schemas.enable` and `value.converter.schemas.enable` properties. The {prodname} {link-prefix}:{link-tutorial}[tutorial] shows what the messages look like when both payload and schemas are included.
Including schemas causes the messages to be very verbose. If you want your messages serialized with JSON, consider setting these properties to `false` to exclude the verbose schema information.
The Avro binary format is compact and efficient, and Avro schemas make it possible to ensure that the messages have the correct structure. Avro's schema evolution mechanism makes it possible to evolve the schemas over time, which is essential for {prodname} connectors that dynamically generate the message schemas to match the structure of the database tables. Over time, the change events captured by {prodname} connectors and written by Kafka Connect into a topic may have different versions of the same schema. Avro serialization makes it easier for consumers to adapt to the changing schema.
Using Avro to serialize message keys and values is a Technology Preview feature. Technology Preview features are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete; therefore, Red Hat does not recommend implementing any Technology Preview features in production environments. This Technology Preview feature provides early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. For more information about support scope, see link:https://access.redhat.com/support/offerings/techpreview/[Technology Preview Features Support Scope].
* An Avro converter that you can configure in Kafka Connect workers. This converter maps Kafka Connect schemas to Avro schemas. The converter then uses the Avro schemas to serialize the message keys and values into Avro's compact binary form.
To use the {registry} with {prodname}, you must add {registry} converters and their dependencies to the Kafka Connect container image that you are using for running {prodname}.
The {registry} project also provides a JSON converter. This converter combines the advantage of less verbose messages with human-readable JSON. Messages do not contain the schema information themselves, but only a schema ID.
. Install the Avro converter from link:https://repo1.maven.org/maven2/io/apicurio/apicurio-registry-distro-connect-converter/{apicurio-version}/apicurio-registry-distro-connect-converter-{apicurio-version}-converter.tar.gz[the installation package] into Kafka Connect's _libs_ directory or directly into a plug-in directory.
. Install the Avro converter by downloading the {prodname} link:https://access.redhat.com/jbossnetwork/restricted/listSoftware.html?product=red.hat.integration&downloadType=distributions[Service Registry Kafka Connect] zip file and extracting it into the {prodname} connector's directory.
In your environment, you might want to use a {prodname} pre-built container to deploy {prodname} connectors that use Avro serializaion. Follow the procedure here to do that. In this procedure, you build a custom Kafka Connect container image for {prodname}, which uses the Avro converter.
This deploys the latest development version of the {registry} operator from the `master` branch. To deploy other versions, specify a different branch or tag, or edit the operator image reference in the file.
+
.. Create a new {registry} deployment by specifying the in-memory persistence option in one of the example custom resources, for example:
The in-memory deployment is not suitable for production. Use the Apache Kafka persistence option for production. For more information, see {LinkServiceRegistryGetStart}[NameServiceRegistryGetStart].
.. Copy link:https://github.com/debezium/debezium-examples/blob/master/tutorial/debezium-with-apicurio/Dockerfile[`Dockerfile`] to a convenient location. This file has the following content:
curl https://repo1.maven.org/maven2/io/apicurio/apicurio-registry-distro-connect-converter/$APICURIO_VERSION/apicurio-registry-distro-connect-converter-$APICURIO_VERSION-converter.tar.gz | tar xzv
.. Download the link:https://access.redhat.com/jbossnetwork/restricted/listSoftware.html?product=red.hat.integration&downloadType=distributions[{registry} Kafka Connect] zip file.
.. Extract the content into the directory that contains the {prodname} connector that you are configuring to use Avro serialization.
.. Create a custom image for Kafka Connect. See link:{LinkCDCInstallOpenShift}[NameCDCInstallOpenShift] for an example of how to do this. Start with the `Dockerfile` in that example. Then add the {registry} converters to the connector directories.
This registers `inventory-connector` and the connector starts to run against the `inventory` database.
.. To verify that the connector was created and has started to monitor the database, follow the steps at the end of the example procedure in link:{LinkCDCGettingStarted}#creating-connector-monitor-inventory-database[NameCDCGettingStarted].
This can lead to problems during serialization if the column name does not also adhere to the Avro naming rules.
Each {prodname} connector provides a configuration property, `sanitize.field.names` that you can set to `true` if you have columns that do not adhere to Avro rules for names. Setting `sanitize.field.names` to `true` allows serialization of non-conformant fields without having to actually modify your schema.
please see the https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mysql-and-the-avro-message-format[MySQL and the Avro message format] tutorial example.
