The Matillion CDC Agent is provided as a Container image. This image is required to be deployed and configured within your organizations private cloud and/or on-premise technology stack and will require access to the source database, the target cloud data lake and secrets management application of either AWS Secrets Manager or Azure Key Vault.
A high-level overview of the installation process:
- A new agent is created in Matillion Data Loader
- A Platform Key is generated (if not already done so for this Organization) and stored in AWS Secrets Manager or Azure Key Vault. Database passwords must also be stored in this manner
- Cloud resources are created for use by the agent such as storage and logs. See specific installation guides for more information:
- The agent is deployed to the appropriate environment with the correct environment variables
Once Matillion has received and added your public key into the CDC service, you will be able to start your agent. If anything is misconfigured, the agent's logs will provide both information and warnings, allowing for potential self correction. If you do encounter any problems that cannot be resolved, please contact Matillion Support.
Once the agent has successfully started, you may go back to the Matillion Data Loader UI and setup a new CDC pipeline.
Important: Your source database will also require configuration to work with CDC. This is independent from the agent installation process - more information can be found in the CDC Sources category. Your choice of agent installation does not limit the available data sources.
It is possible to set up all required resources manually via your cloud platform dashboards. Even if you are wanting to use templates, it is wise to familiarise yourself with the manually installation steps as it can help you understand your final installation as well as aid in setting up the additional manual resources that your template does not cover.
That said, manual installations are long processes with many involved parts. Templates should be used where at all possible and it is always recommended to inform and take advice from your cloud/network administrator before beginning.
Templated Installation (Recommended)
Several types of templates are available that will help provision and configure some of the necessary resources for using the CDC agent. It is highly recommended to use templated installations to lessen the burden on the user. See the documentation for your chosen template to understand which resources are included and what the prerequisites are for installation.
AWS Templates come in two types; Basic and Advanced. Basic templates come with fewer resources for those who have preexisting AWS resources they wish to use in their CDC pipeline. Advanced templates provide a more complete setup and are generally the recommended way to install the CDC agent.
- CloudFormation - Advanced Resources (Recommended)
- CloudFormation - Basic Resources
- Terraform - Advanced Resources
- Terraform - Basic Resources
Azure templates will provision and configure your Azure Containers resources.
Kubernetes templates are provided for users who want to manage their own containers. While the agent itself may not be on AWS or Azure, it does require some resources such as AWS Secrets Manager or Azure Key Vault in order to function and the template should be chosen accordingly.
- The Matillion CDC agent can run only one CDC pipeline. Each CDC pipeline requires a new agent installation.
- An agent entry must be added in Matillion Data Loader, first. Then the agent itself must be installed on your cloud platform.
Container Images cannot be accessed directly via a browser. To pull a container image from the public repository specified, please reference the AWS documentation. The agent container image is available at the following public repository:
The agent requires direct access to the Matillion CDC servers for control and management. To enable this route, the agent will require outbound access to the following public ip addresses:
184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52
Matillion recommends providing the CDC Agent with 4 CPU Cores and 8GB RAM. The agent is intended for use with x86 Linux systems.