DBZ-2282 Misc. clean-up

This commit is contained in:
Gunnar Morling 2020-07-31 15:36:46 +02:00
parent 26ed1bc67c
commit 4bfaa1f949
4 changed files with 53 additions and 8 deletions

View File

@ -1,6 +1,7 @@
Cobolbaby Cobolbaby
Aaron Brady Aaron Brady
Aaron Rosenberg Aaron Rosenberg
Abhishek Gupta
Addison Higham Addison Higham
Adrian Kreuziger Adrian Kreuziger
Akshath Patkar Akshath Patkar

View File

@ -6,6 +6,9 @@ The application itself contains the `core` module and a set of modules responsib
The per-module integration tests depend on the availability of the external services. The per-module integration tests depend on the availability of the external services.
It is thus recommended to execute integration tests per-module and set-up necessary pre-requisities beforehand. It is thus recommended to execute integration tests per-module and set-up necessary pre-requisities beforehand.
Note: running these tests against external infrastructure may incur cost with your cloud provider.
We're not going to pay your AWS/GCP/Azure bill.
## Amazon Kinesis ## Amazon Kinesis
* Execute `aws configure` as described in AWS CLI [getting started](https://github.com/aws/aws-cli#getting-started) guide and setup the account. * Execute `aws configure` as described in AWS CLI [getting started](https://github.com/aws/aws-cli#getting-started) guide and setup the account.
@ -22,17 +25,37 @@ It is thus recommended to execute integration tests per-module and set-up necess
## Azure Event Hubs ## Azure Event Hubs
Login into your Azure account and create a resource group, e.g. on the CLI:
```shell
az login
az group create --name eventhubstest --location westeurope
```
### Create an Event Hubs namespace ### Create an Event Hubs namespace
Create an [Event Hubs namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features?WT.mc_id=debezium-docs-abhishgu#namespace). Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=debezium-docs-abhishgu#create-an-event-hubs-namespace), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=debezium-docs-abhishgu#create-an-event-hubs-namespace) etc. Create an [Event Hubs namespace](https://docs.microsoft.com/azure/event-hubs/event-hubs-features#namespace). Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create#create-an-event-hubs-namespace), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli#create-an-event-hubs-namespace) etc., e.g. on the CLI:
```shell
az eventhubs namespace create --name debezium-test --resource-group eventhubstest -l westeurope
```
### Create an Event Hub ### Create an Event Hub
Create an Event Hub (equivalent to a topic) with `one` partition. Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create?WT.mc_id=debezium-docs-abhishgu#create-an-event-hub), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli?WT.mc_id=debezium-docs-abhishgu#create-an-event-hub) etc. Create an Event Hub (equivalent to a topic) with `one` partition. Check the documentation for options on how do this using the [Azure Portal](https://docs.microsoft.com/azure/event-hubs/event-hubs-create#create-an-event-hub), [Azure CLI](https://docs.microsoft.com/azure/event-hubs/event-hubs-quickstart-cli#create-an-event-hub) etc. , e.g. on the CLI:
```shell
az eventhubs eventhub create --name debezium-test-hub --resource-group eventhubstest --namespace-name debezium-test
```
### Build the module ### Build the module
[Get the Connection string](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=debezium-docs-abhishgu) required to communicate with Event Hubs. The format is: `Endpoint=sb://<NAMESPACE>/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY_VALUE>` [Get the Connection string](https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string) required to communicate with Event Hubs. The format is: `Endpoint=sb://<NAMESPACE>/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY_VALUE>`.
E.g. on the CLI:
```shell
az eventhubs namespace authorization-rule keys list --resource-group eventhubstest --namespace-name debezium-test --name RootManageSharedAccessKey
```
Set environment variables required for tests: Set environment variables required for tests:
@ -44,9 +67,31 @@ export EVENTHUBS_NAME=<name of the Event hub created in previous step>
Execute the tests: Execute the tests:
```shell ```shell
mvn clean install -DskipITs=false -Deventhubs.connection.string=$EVENTHUBS_CONNECTION_STRING -Deventhubs.hub.name=$EVENTHUBS_NAME -am -pl debezium-server-eventhubs mvn clean install -DskipITs=false -Deventhubs.connection.string=$EVENTHUBS_CONNECTION_STRING -Deventhubs.hub.name=$EVENTHUBS_NAME -am -pl :debezium-server-eventhubs
``` ```
### Examine Events in the Event Hub
E.g. using kafkacat. Create _kafkacat.conf_:
```shell
metadata.broker.list=debezium-test.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username=$ConnectionString
sasl.password=Endpoint=sb://debezium-test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<access key>
```
Start consuming events:
export KAFKACAT_CONFIG=<path to kafkacat.conf>
kafkacat -b debezium-test.servicebus.windows.net:9093 -t debezium-test-hub
### Clean up ### Clean up
Delete the Event Hubs namespace Delete the Event Hubs namespace and log out, e.g. on the CLI:
```shell
az group delete -n eventhubstest
az logout
```

View File

@ -25,7 +25,6 @@
<dependency> <dependency>
<groupId>com.azure</groupId> <groupId>com.azure</groupId>
<artifactId>azure-messaging-eventhubs</artifactId> <artifactId>azure-messaging-eventhubs</artifactId>
<version>5.1.1</version>
</dependency> </dependency>
<!-- Testing --> <!-- Testing -->

View File

@ -445,7 +445,7 @@ This is not supported by Pulsar so a surrogate key must be used.
==== Azure Event Hubs ==== Azure Event Hubs
https://docs.microsoft.com/azure/event-hubs/event-hubs-about?WT.mc_id=debezium-docs-abhishgu[Azure Event Hubs] is a big data streaming platform and event ingestion service that can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters. https://docs.microsoft.com/azure/event-hubs/event-hubs-about[Azure Event Hubs] is a big data streaming platform and event ingestion service that can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.
[cols="35%a,10%a,55%a",options="header"] [cols="35%a,10%a,55%a",options="header"]
|======================= |=======================
@ -459,7 +459,7 @@ https://docs.microsoft.com/azure/event-hubs/event-hubs-about?WT.mc_id=debezium-d
|[[connection-string]]<<connection-string, `debezium.sink.eventhubs.connectionstring`>> |[[connection-string]]<<connection-string, `debezium.sink.eventhubs.connectionstring`>>
| |
|https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string?WT.mc_id=debezium-docs-abhishgu[Connection string] required to communicate with Event Hubs. The format is: `Endpoint=sb://<NAMESPACE>/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY_VALUE>` |https://docs.microsoft.com/azure/event-hubs/event-hubs-get-connection-string[Connection string] required to communicate with Event Hubs. The format is: `Endpoint=sb://<NAMESPACE>/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY_VALUE>`
|[[hub-name]]<<hub-name, `debezium.sink.eventhubs.hubname`>> |[[hub-name]]<<hub-name, `debezium.sink.eventhubs.hubname`>>
| |