Contents x
Deploying a CDC agent in AWS quick guide
  • Dark
    Light
Contents

Deploying a CDC agent in AWS quick guide

  • Dark
    Light

Overview

Use this guide to add a CDC agent in Matillion Data Loader and then deploy that agent in Amazon Web Services (AWS). Creating and deploying an agent are required steps to set up a CDC pipeline in Matillion Data Loader.

Note

For best performance, your AWS region should be geographically similar to your Matillion Hub account region.

Create a CDC agent in Matillion Data Loader

  1. Log in to the Matillion Hub.
  2. The My Accounts lists any accounts you have already created or joined. At the bottom of this list, click Add new account. Read Create an Account to learn more about this topic.
Note

Each Matillion Hub account can generate its own unique platform key that your CDC agent will use to communicate with Matillion Data Loader. With this in mind, create the CDC agent in the account that matches the platform key you will be using.

  1. Choose Matillion Data Loader as a service on the Select your service page.
  2. On the Matillion Data Loader dashboard, scroll to the lower-right of the UI and choose your region.
  3. Select Agents in the left sidebar and click Add agent.
  4. Give your agent a sensible Agent name and Description. Click Continue.
  5. Since this guide is for AWS, select AWS as your cloud provider.
  6. Choose CloudFormation as the service to provision and deploy your cloud resources from for the CDC agent installation.
  7. In the Prerequisites for agent setup, note the following values:
    • ID_ORGANIZATION: This value is used when deploying the CDC agent in AWS. The value is unique per agent.
    • ID_AGENT: Also used when deploying the CDC agent. The value is unique per agent.
    • PLATFORM_WEBSOCKET_ENDPOINT: Also used when deploying the Agent. The value is unique for the Matillion Data Loader region (US or EU).
    • Public/Private key pair: This is a generated value. If you haven't generated a platform secret for your account yet, Matillion Data Loader will prompt you to do so when creating a CDC pipeline. You need to store this value in AWS Secrets Manager where your CDC agent can access it. For security reasons, this key pair can only be generated and shown once per account, so make sure to copy and save it for future use.
Note

You can revisit this page if required.

  1. Check the I have saved the private key in AWS Secrets Manager and made a note of the secret name checkbox.
  2. Click Submit key pair.

Deploy the CDC Agent in AWS

You can deploy a CDC agent in multiple ways. However, this guide follows the CloudFormation ECS Fargate Advanced Template method.

  1. Download the CloudFormation template from the Matillion Data Loader documentation.
  2. Log in to the AWS console.
  3. In the AWS console, navigate to the region dropdown and select the region in which you wanted to deploy the CDC agent.
Note

Make sure the region is either eu or us, depending on the Matillion Data Loader region you are building the pipeline within. In the AWS console, you must choose the same region.

  1. Navigate to CloudFormation and click Create stack > With new resources (standard).

  2. Under Specify template, select Upload a template file. Select Choose file and upload the CloudFormation template from step 1. Click Next.

  3. Provide the following information for the stack details. Note that the template you choose will autofill part of the information.

    • Stack Name: A unique name for the stack.
    • AgentID: The value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
    • Assign Public: Set to Enabled.
    • Bucket: The name of the S3 bucket for staging. The template should autofill this value.
    • ClusterName: The name for the ECS Fargate cluster to be created for hosting your agent. The template should autofill this value.
    • ExecutionRoleName: The role name for the role that ECS will use to initialize the task.
    • ImageUrl: The URL of the repository you are pulling the CDC agent image from. The template should autofill this value.
    • LogGroupName: Name of the CloudWatch Log Group for agent logging. The template should autofill this value.
    • OrganizationID: This is the value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
    • PlatformKeyName: The name of the AWS Secrets Manager secret in which your generated private key is stored. Consult the AWS Secrets Manager documentation for more information about creating a secrets manager in AWS.
    • PlatformWebSocketEndpoint: The value for PLATFORM_WEBSOCKET_ENDPOINT, copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
    • Region: The region for these resources to be created in, either eu or us. Eg: eu-west-1 if you have chosen eu.
    • RoleName: The role name for the role that the task will run as. The template should autofill this value.
    • SecretARNs: The Secret ARN value of the secret created in step 9 of Create a CDC Agent in Matillion Data Loader using AWS Secrets Manager. You can also leave this field empty with an asterisk character.
    • SecurityGroups: If there are any existing security groups that you have, select them from the drop-down menu. Otherwise, create a new security group with required outbound rules as mentioned below.
      • When you first create a security group, it has an outbound rule that allows all outbound traffic from the resource. You can remove the rule and add outbound rules that allow specific outbound traffic only. If your security group has no outbound rules, no outbound traffic is allowed.
      • When you add inbound rules for ports 22 (SSH) or 3389 (RDP) so that you can access your EC2 instances, authorize only specific IP address ranges. If you specify 0.0.0.0/0 (IPv4) and :/ (IPv6), this enables anyone to access your instances from any IP address using the specified protocol.
    • ServiceName: The name of the ECS Service that you want these tasks to run under. The template should autofill this value.
    • SubnetIDs: If you have any existing Subnet IDs that can be used, you can select one from the drop-down menu or else you can create a new one.
      • When you create a subnet, you specify its IP addresses. Depending on the configuration of the VPC, set IPv4 only or IPv6 only.
      • Each subnet must be associated with a route table, which specifies the allowed routes for outbound traffic leaving the subnet.
      • Each subnet must be associated with a network ACL. Every subnet that you create is automatically associated with the default network ACL for the VPC.
    • TaskDefinitionName: The name of the task definition for agent tasks.

  4. Click Next.

  5. Review the information you've entered, click the required check boxes and click Create Stack.

  6. The stack creation will then begin and should complete in approximately five minutes.

  7. When the stack creation is complete, the agent container will be deployed as an AWS Elastic Container Service (ECS) Cluster. The CloudFormation Template also creates resources in IAM, S3, and CloudWatch Logs.

In Matillion Data Loader, your created CDC agent's status should display as Connected and offer the Add Pipeline button.

What's Next

Cookie consent

We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content based on your browsing and navigation history.