Deploying a CDC Agent in Azure using the CLI

Overview

Use this guide to add a CDC agent in Matillion Data Loader and deploy the agent using the Azure command line interface (CLI). Creating and deploying an agent are required steps before you can set up a CDC pipeline in Matillion Data Loader.

Prerequisites

Complete the Creating a CDC Agent in Matillion Data Loader and Azure Prerequisite Steps sections in Deploying a CDC agent in Azure quick guide.

Deploy your CDC agent using the Azure CLI

1. Export the following variables from your Azure environment (see the variable, value, and description):

RESOURCE_GROUP
your-resource-group-name
Target resource group. Should be the same group containing your target Vnet and Subnet if using private networking (recommended).

CONTAINER_NAME
desired-container-name
Provide a descriptive name for the container.

CONTAINER_IMAGE
public.ecr.aws/matillion/cdc-agent:2
Public agent image path.

CONTAINER_CPU
4
We recommend 4 vCPU.

CONTAINER_RAM
8
We recommended 8 GB RAM.

AZURE_LOCATION
For example, southcentralus
Target location name.

AZURE_VNET_NAME
For example, vnet-southcentralus
Target Vnet, if using private networking.

AZURE_SUBNET
For example, subnet-cdc-containers
Target subnet, if using private networking.

LOG_ANALYTICS_WORKSPACE
For example, azure-us-mssql-blob-cli
Target Log Analytics workspace name—for shipping logs.

LOG_ANALYTICS_WORKSPACE_KEY
For example, lYfsWJqa6J0q9+PDDLMqyjk5iXXXXXXXXXXXgcIzz4sBrsZ7/n9C8eSoqnyRong8Uxag4yfw==
Target Log Analytics workspace name—for shipping logs.

AZURE_CLIENT_ID
For example, 72de73c2-XXXX-XXXX-9fcf-c14b20d34bb1
App Registration's Application/Client ID.

ID_ORGANIZATION
For example, b4a525af-XXXX-XXXX-7c1f-ef898f4d6d49
Hub Organization ID.

ID_AGENT
For example, 91470067-XXXX-XXXX-963d-588a1f729ee3
Hub Agent ID.

PLATFORM_WEBSOCKET_ENDPOINT
For example, wss://ws-eu.matillion-cdc-prod.matillion.com:443/ws
URL of the Hub Websocket endpoint.

AZURE_SECRET_KEY_VAULT_URL
https://your-key-vault-name.vault.azure.net/
URL of Key Vault containing Hub key.

AZURE_TENANT_ID
For example, 09d19996-XXXX-XXXX-8332-37120f9bba10
Hub Directory/Tenant ID.

PLATFORM_KEY_NAME
For example, agent-rsa
Name of the key vault secret containing the Hub key.

PLATFORM_KEY_PROVIDER
azure-key-vault
Static value.

SECRET_PROVIDERS
azure-key-vault:1
Static value.

AZURE_CLIENT_SECRET
For example, RbB8Q~EfgoqLG21E-XXXXXXXX~Z9B4em6I6bdt
App Registration's client secret value (not secret ID).

AZURE_STORAGE_KEY
For example, FOHf+HsmBATWsdMa5f97BAvIx50XXXXXXXXXXXDlaY3nBrSrpB+RnDYY7+E4jdNCQYkm3A==
Key of target storage account.

2. Open the Azure CLI and run the following command:

az container create \
    --resource-group $RESOURCE_GROUP \
    --name $CONTAINER_NAME \
    --image $CONTAINER_IMAGE \
    --os-type Linux \
    --cpu $CONTAINER_CPU \
    --memory $CONTAINER_RAM \
    --location $AZURE_LOCATION \
    --restart-policy Always \
    --environment-variables AZURE_CLIENT_ID=$AZURE_CLIENT_ID ID_ORGANIZATION=$ID_ORGANIZATION ID_AGENT=$ID_AGENT PLATFORM_WEBSOCKET_ENDPOINT=$PLATFORM_WEBSOCKET_ENDPOINT AZURE_SECRET_KEY_VAULT_URL=$AZURE_SECRET_KEY_VAULT_URL AZURE_TENANT_ID=$AZURE_TENANT_ID PLATFORM_KEY_NAME=$PLATFORM_KEY_NAME PLATFORM_KEY_PROVIDER=$PLATFORM_KEY_PROVIDER SECRET_PROVIDERS=$SECRET_PROVIDERS \
    --secure-environment-variables AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY \
    --ip-address Private \
    --vnet $AZURE_VNET_NAME \
    --subnet $AZURE_SUBNET \
    --ports 8080 \
    --protocol TCP \
    --log-analytics-workspace $LOG_ANALYTICS_WORKSPACE \
    --log-analytics-workspace-key $LOG_ANALYTICS_WORKSPACE_KEY

In Matillion Data Loader, your created CDC agent's status should display as Connected and offer the Add Pipeline button.