CDC Agent Installation
The Matillion CDC Agent is provided as a container image. This image is required to be deployed and configured within your organization's private cloud and/or on-premises technology stack, and will require access to the source database, the target cloud data lake, and secrets management application of either AWS Secrets Manager, Azure Key Vault, or Google Secret Manager.
A high-level overview of the installation process
- A new agent is defined within Matillion Data Loader, which provides the required values for deployment.
- A Platform Key is generated (if not already done so for this account) and stored in AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. Database passwords must also be stored in this manner.
- Cloud resources are created for use by the agent such as storage and logs. See specific installation guides for more information:
- The agent is deployed to the appropriate environment with the correct environment variables.
Once Matillion has received and added your public key into the CDC service, you will be able to start your agent. If anything is configured wrong, the agent's logs will provide both information and warnings, allowing for potential self correction. If you do encounter any problems that cannot be resolved, please contact Matillion Support.
Once the agent has successfully started, you may go back to the Matillion Data Loader UI and setup a new CDC pipeline.
Your source database will also require configuration to work with CDC. This is independent from the agent installation process - more information can be found in the CDC Sources category. Your choice of agent installation does not limit the available data sources.
It is possible to set up all required resources manually via your cloud platform dashboards. If you wish to use templates, it is recommended to familiarize yourself with the manual installation steps, as it can help you understand your final installation as well as aid in setting up the additional manual resources that your template does not cover.
However, manual installations are long processes with many involved parts. Templates should be used where at all possible and it is always recommended to inform and take advice from your cloud/network administrator before beginning.
Templated Installation (Recommended)
Several types of templates are available that will help provision and configure some of the necessary resources for using the CDC agent. It's highly recommended to use templated installations to lessen the burden on the user. See the documentation for your chosen template to understand which resources are included and what the prerequisites are for installation.
AWS Templates come in two types: Basic and Advanced.
Basic templates include less resources and are for users who have pre-existing AWS resources that you want to use in your CDC pipelines. Advanced templates give you a more robust setup and are the recommended method for installing the CDC agent.
- CloudFormation - Advanced Resources (Recommended)
- CloudFormation - Basic Resources
- Terraform - Advanced Resources
- Terraform - Basic Resources
Azure templates will provision and configure your Azure Containers resources.
Kubernetes templates are provided for users who want to manage their own containers. While the agent itself may not be on AWS or Azure, it does require some resources such as AWS Secrets Manager, Azure Key Vault, or Google Secret Manager to function, and the template should be chosen accordingly.
The GCP templates will provision and configure your Google Cloud Containers resources.
- The Matillion CDC agent can run only one CDC pipeline. Each CDC pipeline requires a new agent installation.
- An agent entry must be added in Matillion Data Loader, first. Then the agent itself must be installed on your cloud platform.
- CDC supports multiple schema and you can select any schemas and tables that you wish to include in the CDC pipeline.
Container images can't be accessed directly via a browser. To pull a container image from the public repository specified, please reference the AWS documentation. The agent container image is available at the following public repository:
The agent requires direct access to the Matillion CDC servers for control and management. To enable this route, the agent will require outbound access to the following public IP addresses:
|IP address||Port number|
Matillion recommends providing the CDC Agent with 4 CPU Cores and 8 GB RAM. The agent is intended for use with x86 Linux systems.