Agent (Azure) Template
  • Dark
    Light

Agent (Azure) Template

  • Dark
    Light

Overview

It is possible to use a Kubernetes platform to manage CDC agent containers. The provided Kubernetes pod templates will allow the agent image to be retrieved from Amazon ECR and deployed to any compatible x86 Linux Kubernetes platform.

See the official documentation on Kubernetes pods for more information.

It should be understood that using the Matillion CDC Agent in a Kubernetes platform is among the more manual methods of setting up a Matillion Data Loader CDC pipeline with a great burden of knowledge on the user. This should only be attempted by users who are familiar with both Kubernetes and Azure.

Prerequisites

Resources

This template is intended for users who are accustomed to setting up their own Kubernetes platforms and offers a degree of freedom that other templates (such as ARM) do not. As such, we cannot dictate which resources you may choose to use to build your CDC stack.

However, this template does require Azure Key Vault resources and permissions to the agent image on the ECR Public Gallery.

  • Azure Key Vault secrets for the following:

    • Platform Key
    • Database passwords
  • IAM Role for Task Execution:

    • ecr-public:GetAuthorizationToken
    • sts:GetServiceBearerToken

See the AWS documentation for more information on images from the ECR Public Gallery.

Azure Template

Parameters

The following environment variables are required by the template. This template requires you to fill in environment variables that allow Matillion Data Loader to identity the agent.

Environment Variable Description
ID_ORGANIZATION This is provided to you by the Matillion Data Loader client when setting up a new agent.
ID_AGENT This is provided to you by the Matillion Data Loader client when setting up a new agent.
PLATFORM_KEY_NAME The name of the key storing your platform secret that is generated the first time you attempt to create an agent.
PLATFORM_KEY_PROVIDER The service that supplies your platform key. This must be azure-key-vault for the Azure Template.
PLATFORM_WEBSOCKET_ENDPOINT This value must be set to wss://ws-<region>.matillion-cdc-prod.matillion.com:443/ws where <region> is either eu or us depending on the Data Loader region you are building the pipeline in.
SECRET_PROVIDERS The service that holds your database passwords. This must be azure-key-vault:1 for the Azure Template.
AZURE_SECRET_KEY_VAULT_URL The URL of your azure key vault. e.g. https://<vault-name>.vault.azure.net

Download


What's Next