From 4bfaa1f949e3942b37814ac855f8acd4ae6b7107 Mon Sep 17 00:00:00 2001 From: Gunnar Morling Date: Fri, 31 Jul 2020 15:36:46 +0200 Subject: [PATCH] DBZ-2282 Misc. clean-up --- COPYRIGHT.txt | 1 + debezium-server/README.md | 55 +++++++++++++++++-- .../debezium-server-eventhubs/pom.xml | 1 - .../pages/operations/debezium-server.adoc | 4 +- 4 files changed, 53 insertions(+), 8 deletions(-) diff --git a/COPYRIGHT.txt b/COPYRIGHT.txt index 0a3bc8f16..1fe8c48bd 100644 --- a/COPYRIGHT.txt +++ b/COPYRIGHT.txt @@ -1,6 +1,7 @@ Cobolbaby Aaron Brady Aaron Rosenberg +Abhishek Gupta Addison Higham Adrian Kreuziger Akshath Patkar diff --git a/debezium-server/README.md b/debezium-server/README.md index e2c3c1512..9ea807500 100644 --- a/debezium-server/README.md +++ b/debezium-server/README.md @@ -6,6 +6,9 @@ The application itself contains the `core` module and a set of modules responsib The per-module integration tests depend on the availability of the external services. It is thus recommended to execute integration tests per-module and set-up necessary pre-requisities beforehand. +Note: running these tests against external infrastructure may incur cost with your cloud provider. +We're not going to pay your AWS/GCP/Azure bill. + ## Amazon Kinesis * Execute `aws configure` as described in AWS CLI [getting started](https://github.com/aws/aws-cli#getting-started) guide and setup the account. @@ -22,17 +25,37 @@ It is thus recommended to execute integration tests per-module and set-up necess ## Azure Event Hubs +Login into your Azure account and create a resource group, e.g. on the CLI: + +```shell +az login +az group create --name eventhubstest --location westeurope +``` + ### Create an Event Hubs namespace -Create an [Event Hubs namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features?WT.mc_id=debezium-docs-abhishgu#namespace). Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=debezium-docs-abhishgu#create-an-event-hubs-namespace), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=debezium-docs-abhishgu#create-an-event-hubs-namespace) etc. +Create an [Event Hubs namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features#namespace). Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create#create-an-event-hubs-namespace), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli#create-an-event-hubs-namespace) etc., e.g. on the CLI: + +```shell +az eventhubs namespace create --name debezium-test --resource-group eventhubstest -l westeurope +``` ### Create an Event Hub -Create an Event Hub (equivalent to a topic) with `one` partition. Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=debezium-docs-abhishgu#create-an-event-hub), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=debezium-docs-abhishgu#create-an-event-hub) etc. +Create an Event Hub (equivalent to a topic) with `one` partition. Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create#create-an-event-hub), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli#create-an-event-hub) etc. , e.g. on the CLI: + +```shell +az eventhubs eventhub create --name debezium-test-hub --resource-group eventhubstest --namespace-name debezium-test +``` ### Build the module -[Get the Connection string](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=debezium-docs-abhishgu) required to communicate with Event Hubs. The format is: `Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey=` +[Get the Connection string](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string) required to communicate with Event Hubs. The format is: `Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey=`. +E.g. on the CLI: + +```shell +az eventhubs namespace authorization-rule keys list --resource-group eventhubstest --namespace-name debezium-test --name RootManageSharedAccessKey +``` Set environment variables required for tests: @@ -44,9 +67,31 @@ export EVENTHUBS_NAME= Execute the tests: ```shell -mvn clean install -DskipITs=false -Deventhubs.connection.string=$EVENTHUBS_CONNECTION_STRING -Deventhubs.hub.name=$EVENTHUBS_NAME -am -pl debezium-server-eventhubs +mvn clean install -DskipITs=false -Deventhubs.connection.string=$EVENTHUBS_CONNECTION_STRING -Deventhubs.hub.name=$EVENTHUBS_NAME -am -pl :debezium-server-eventhubs ``` +### Examine Events in the Event Hub + +E.g. using kafkacat. Create _kafkacat.conf_: + +```shell +metadata.broker.list=debezium-test.servicebus.windows.net:9093 +security.protocol=SASL_SSL +sasl.mechanisms=PLAIN +sasl.username=$ConnectionString +sasl.password=Endpoint=sb://debezium-test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey= +``` + +Start consuming events: + +export KAFKACAT_CONFIG= +kafkacat -b debezium-test.servicebus.windows.net:9093 -t debezium-test-hub + ### Clean up -Delete the Event Hubs namespace \ No newline at end of file +Delete the Event Hubs namespace and log out, e.g. on the CLI: + +```shell +az group delete -n eventhubstest +az logout +``` diff --git a/debezium-server/debezium-server-eventhubs/pom.xml b/debezium-server/debezium-server-eventhubs/pom.xml index 645dd8a39..3c41816d4 100644 --- a/debezium-server/debezium-server-eventhubs/pom.xml +++ b/debezium-server/debezium-server-eventhubs/pom.xml @@ -25,7 +25,6 @@ com.azure azure-messaging-eventhubs - 5.1.1 diff --git a/documentation/modules/ROOT/pages/operations/debezium-server.adoc b/documentation/modules/ROOT/pages/operations/debezium-server.adoc index f121c48a3..fab6068c5 100644 --- a/documentation/modules/ROOT/pages/operations/debezium-server.adoc +++ b/documentation/modules/ROOT/pages/operations/debezium-server.adoc @@ -445,7 +445,7 @@ This is not supported by Pulsar so a surrogate key must be used. ==== Azure Event Hubs -https://docs.microsoft.com/azure/event-hubs/event-hubs-about?WT.mc_id=debezium-docs-abhishgu[Azure Event Hubs] is a big data streaming platform and event ingestion service that can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters. +https://docs.microsoft.com/azure/event-hubs/event-hubs-about[Azure Event Hubs] is a big data streaming platform and event ingestion service that can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters. [cols="35%a,10%a,55%a",options="header"] |======================= @@ -459,7 +459,7 @@ https://docs.microsoft.com/azure/event-hubs/event-hubs-about?WT.mc_id=debezium-d |[[connection-string]]<> | -|https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=debezium-docs-abhishgu[Connection string] required to communicate with Event Hubs. The format is: `Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey=` +|https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string[Connection string] required to communicate with Event Hubs. The format is: `Endpoint=sb:///;SharedAccessKeyName=;SharedAccessKey=` |[[hub-name]]<> |