CDC Agent Installation
  • Dark
    Light

CDC Agent Installation

  • Dark
    Light

Overview

The Matillion CDC Agent is provided as a container image. This image is required to be deployed and configured within your organization's private cloud and/or on-premises technology stack, and will require access to the source database, the target cloud data lake, and secrets management application of either AWS Secrets Manager or Azure Key Vault.


A high-level overview of the installation process

  1. A new agent is created in Matillion Data Loader.
  2. A Platform Key is generated (if not already done so for this account) and stored in AWS Secrets Manager or Azure Key Vault. Database passwords must also be stored in this manner.
  3. Cloud resources are created for use by the agent such as storage and logs. See specific installation guides for more information:
  4. The agent is deployed to the appropriate environment with the correct environment variables.

Once Matillion has received and added your public key into the CDC service, you will be able to start your agent. If anything is configured wrong, the agent's logs will provide both information and warnings, allowing for potential self correction. If you do encounter any problems that cannot be resolved, please contact Matillion Support.

Once the agent has successfully started, you may go back to the Matillion Data Loader UI and setup a new CDC pipeline.

Your source database will also require configuration to work with CDC. This is independent from the agent installation process - more information can be found in the CDC Sources category. Your choice of agent installation does not limit the available data sources.


Manual installation

It is possible to set up all required resources manually via your cloud platform dashboards. If you wish to use templates, it is recommended to familiarize yourself with the manual installation steps, as it can help you understand your final installation as well as aid in setting up the additional manual resources that your template does not cover.

However, manual installations are long processes with many involved parts. Templates should be used where at all possible and it is always recommended to inform and take advice from your cloud/network administrator before beginning.


Several types of templates are available that will help provision and configure some of the necessary resources for using the CDC agent. It's highly recommended to use templated installations to lessen the burden on the user. See the documentation for your chosen template to understand which resources are included and what the prerequisites are for installation.

AWS Fargate

AWS Templates come in two types: Basic and Advanced.

Basic templates include less resources and are for users who have pre-existing AWS resources that you want to use in your CDC pipelines. Advanced templates give you a more robust setup and are the recommended method for installing the CDC agent.

Azure Containers

Azure templates will provision and configure your Azure Containers resources.

Kubernetes

Kubernetes templates are provided for users who want to manage their own containers. While the agent itself may not be on AWS or Azure, it does require some resources such as AWS Secrets Manager or Azure Key Vault to function, and the template should be chosen accordingly.

GCP Templates

The GCP templates will provision and configure your Google Cloud Containers resources.


The Agent

  • The Matillion CDC agent can run only one CDC pipeline. Each CDC pipeline requires a new agent installation.
  • An agent entry must be added in Matillion Data Loader, first. Then the agent itself must be installed on your cloud platform.

Location

Container images can't be accessed directly via a browser. To pull a container image from the public repository specified, please reference the AWS documentation. The agent container image is available at the following public repository:

public.ecr.aws/matillion/cdc-agent:2

Access

The agent requires direct access to the Matillion CDC servers for control and management. To enable this route, the agent will require outbound access to the following public IP addresses:

IP address Port number
15.197.249.56 443
3.33.208.120 443
13.248.238.58 443
76.223.106.122 443
75.2.56.46 443
99.83.216.99 443

Technical Requirements

Matillion recommends providing the CDC Agent with 4 CPU Cores and 8 GB RAM. The agent is intended for use with x86 Linux systems.


What's Next