Contents x
Agent Setup UI (CDC)
  • Dark
    Light
Contents

Agent Setup UI (CDC)

  • Dark
    Light

Overview

To run a CDC pipeline, you need to create an agent container. The Matillion CDC Agent is provided as a container image. This image must be deployed and configured within your organization's private cloud or on-premise technology stack, and will require access to the source database, the target cloud data lake, and either AWS Secrets Manager, Azure Key Vault, or Google Secret Manager.

Note

For more information about the agent deployment, please read CDC Agent Installation.

Prerequisites

  • Matillion recommends you set up your database infrastructure for CDC before proceeding with agent installation instructions.
  • Ensure you have the required database authority and privileges to manage datastores.
  • If you aren't a database administrator, contact your system administrator to request access to create a stack in your infrastructure environment.

Create an agent

Follow these instructions to create a CDC agent container through Matillion Data Loader.

  1. From the Matillion Data Loader dashboard, click Agents in the sidebar.
  2. On the Agent containers page, click Add agent.
  3. Provide a unique name for the agent and give a brief description, then click Continue.
  4. Choose a cloud provider for the infrastructure where you will create and deploy your agent container.
  5. Depending on your selection of cloud provider, you will get options to choose a service to provision and deploy your cloud resources.
Note
  • These services help you configure, provision, and manage your cloud resources by treating infrastructure as code.
  • For detailed information about using these services for creating an agent container, please read CDC Agent Installation.
  1. After you choose your service, you will be redirected to a Prerequisite for agent setup page with detailed instructions on how to set up your agent for the selected cloud services.
Note
  • The agent requires direct access to the Matillion CDC servers for control and management.
  • If you do not have access to the CDC product, please contact Matillion for support as you will be unable to continue with the installation process.
  • When creating a new agent through Matillion Data Loader, an identity value (agent_id) will have been automatically generated along with installation instructions. The id_agent and your id_organization will need to be provided as environment variables when deploying the agent.
  • Access to the appropriate secrets management service will need to be granted to the identity the agent container is using. This is typically achieved through IAM roles, depending on your cloud provider.
  • The Matillion CDC agent expects to find your platform secret in either Azure Key Vault or AWS Secrets Manager, depending on where the agent is installed. For more information, refer to Platform Keys.
  1. Once you have successfully created an agent container, click Return to agent list. Locate your new agent in the list and ensure the Agent status is Connected. You can now add a pipeline to the agent.
Warning

You can delete an agent from the agent list at any time, but you must also delete the corresponding container from your infrastructure.

Agent dashboard

When you click the Agents icon in Matillion Data Loader, the Agent containers list is displayed. On this page you can view the list of agents you have created. Refer to the image and description below to familiarize yourself with the components of the agent list view.

Agent list view

The following table describes each of these components:

Page component Title Description
1 Add agent Click to create a new agent.
2 Search Search for agents by typing a partial or complete name of the agent.
3 Filter Filter the list of agents by Pipeline status (in the first dropdown field) and Agent status (in the second dropdown field. The default is to list all agents.
4 Agent List Displays the list of agents created by the current user. You can sort the list by any column by clicking the arrows in the column header (the default is sorted by Agent name), and filter by using the Search and Filter fields above the list.
5 Add pipeline Any agent with a status of Connected which does not have a pipeline assigned will display the Add pipeline button. Click to add a pipeline to the agent. Read CDC Pipelines for details.
6 Agent setup Click this icon to go through the agent set up process described in Creating an agent, above.
7 More actions Click ... to see other actions you can perform on the agent. Edit Agent allows you to change the agent's name and description. Remove Agent deletes the agent from the list but doesn't delete the corresponding container from your infrastructure. See note above.

Agent status

Every agent will have one of these statuses:

  • Disconnected: The agent is not connected to Matillion's servers. You won't be able to add a pipeline to the agent, and any existing pipeline will not run.
  • Connected: The agent is connected to Matillion's servers. A pipeline can be added and will operate normally.

Useful links

What's Next

Cookie consent

We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content based on your browsing and navigation history.