CDC Agent Installation

Overview

The Matillion CDC Agent is provided as a container image. This image is required to be deployed and configured within your organization's private cloud and/or on-premises technology stack, and will require access to the source database, the target cloud data lake, and secrets management application of either AWS Secrets Manager, Azure Key Vault, or Google Secret Manager.

A high-level overview of the installation process

1. A new agent is defined within Matillion Data Loader, which provides the required values for deployment.
2. A Platform Key is generated (if not already done so for this account) and stored in AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. Database passwords must also be stored in this manner.
3. Cloud resources are created for use by the agent such as storage and logs. See specific installation guides for more information:
   • AWS
   • Azure
   • GCP
4. The agent is deployed to the appropriate environment with the correct environment variables.

Once Matillion has received and added your public key into the CDC service, you will be able to start your agent. If anything is configured wrong, the agent's logs will provide both information and warnings, allowing for potential self correction. If you do encounter any problems that cannot be resolved, please contact Matillion Support.

Once the agent has successfully started, you may go back to the Matillion Data Loader UI and setup a new CDC pipeline.

Manual installation

It is possible to set up all required resources manually via your cloud platform dashboards. If you wish to use templates, it is recommended to familiarize yourself with the manual installation steps, as it can help you understand your final installation as well as aid in setting up the additional manual resources that your template does not cover.

However, manual installations are long processes with many involved parts. Templates should be used where at all possible and it is always recommended to inform and take advice from your cloud/network administrator before beginning.

• AWS
• Azure
• GCP

Templated Installation (Recommended)

Several types of templates are available that will help provision and configure some of the necessary resources for using the CDC agent. It's highly recommended to use templated installations to lessen the burden on the user. See the documentation for your chosen template to understand which resources are included and what the prerequisites are for installation.

AWS Fargate

AWS Templates come in two types: Basic and Advanced.

Basic templates include less resources and are for users who have pre-existing AWS resources that you want to use in your CDC pipelines. Advanced templates give you a more robust setup and are the recommended method for installing the CDC agent.

• CloudFormation - Advanced Resources (Recommended)
• CloudFormation - Basic Resources
• Terraform - Advanced Resources
• Terraform - Basic Resources

Azure Containers

Azure templates will provision and configure your Azure Containers resources.

• ARM Template
• Terraform

Kubernetes

Kubernetes templates are provided for users who want to manage their own containers. While the agent itself may not be on AWS or Azure, it does require some resources such as AWS Secrets Manager, Azure Key Vault, or Google Secret Manager to function, and the template should be chosen accordingly.

• AWS Resources
• Azure Resources

GCP Templates

The GCP templates will provision and configure your Google Cloud Containers resources.

• Google Compute Engine
• GKE (Kubernetes)

The Agent

• The Matillion CDC agent can run only one CDC pipeline. Each CDC pipeline requires a new agent installation.
• An agent entry must be added in Matillion Data Loader, first. Then the agent itself must be installed on your cloud platform.
• CDC supports multiple schema and you can select any schemas and tables that you wish to include in the CDC pipeline.

Location

Container images can't be accessed directly via a browser. To pull a container image from the public repository specified, please reference the AWS documentation. The agent container image is available at the following public repository:

public.ecr.aws/matillion/cdc-agent:2

Access

The agent requires direct access to the Matillion CDC servers for control and management. To enable this route, the agent will require outbound access to the following public IP addresses:

Technical Requirements

Matillion recommends providing the CDC Agent with 4 CPU Cores and 8 GB RAM. The agent is intended for use with x86 Linux systems.