Contents x
CloudFormation ECS Fargate Advanced Template
  • Dark
    Light
Contents

CloudFormation ECS Fargate Advanced Template

  • Dark
    Light

Overview

This article details how to install the Matillion CDC Agent on Amazon ECS Fargate using a CloudFormation template.

These steps can only be taken after completing the preliminary steps found in AWS Installation: Templated Installations and assumes you have done so.

There are three options to create a CloudFormation stack in the AWS console for agent deployment.

  1. Quick-create, which will pre-fill values into the template for you. This is the recommended option.
  2. Quick-create link from the docs (it will provide the template but no values will be known to pass).
  3. Download the template yourself and make any tweaks needed.

Please ensure you have:

  • Permissions to provision cloud resources in the AWS environment that the source database is running.
  • Enabled CDC in your source database.
  • Whitelisted the IP address.
  • Set up the template variables as referenced in the documentation.

It is highly recommended you read the documentation and prerequisites before beginning this process. It's likely you will require input from your organization's cloud administrator for access and permissions.

Created Resources

This template will create the following resources in your AWS account:

  • ECS Task Definition.
  • ECS Fargate Cluster.
  • S3 Bucket.
  • CloudWatch Log Group.
  • IAM Roles with permissions for the following:
  • Task Roles to your S3 bucket, ECS, EC2, and Secrets Manager (see "Task IAM Role" section).
  • Task Execution Roles to pull from ECR Public repositories (see "Task Execution IAM Role" section).

Visualization

image.png

Prerequisites

Edit the template

Users should inspect the template in a text editor and ensure the values are as expected before proceeding. In particular, the PLATFORM_WEBSOCKET_ENDPOINT environment variable should be edited to match the expected endpoint and region. Read Environment Variables for more information.

Resources

The advanced template assumes you have certain resources already set up in your AWS stack. You need to provide details on these resources such as names, paths, and ARNs.

  • Subnets within a customer private cloud with outbound access to the relevant data sources and destinations, as well as to ECR and to the Matillion CDC websocket endpoint.
  • Security group controlling ingress/egress within the subnets.
  • Secrets Manager entries for your Platform Key and database passwords.

User Access

You as a user are also expected to have access to certain details and permissions:

  • Access to the Matillion Hub account and Matillion Data Loader.
  • CDC Agent environment variables (generated in Matillion Data Loader when creating a new agent).
  • Matillion Data Loader platform key (generated once per Matillion Data Loader account the first time you make an agent).
  • Access to AWS with the ability to create a stack on a billable account. You may require an administrator from your organization to either give access or perform this process with you.

Create Agent Stack

You should have your Platform Key and Agent Identity environment variables ready before beginning this process. If you do not have these, please first consult the Creating Agents documentation.

Note

You will require permissions to create and manage resources in your AWS account. Please contact your administrator. Matillion's provided templates will not work if you have insufficient permissions.

Using the CloudFormation Download Template

  1. Download the ECS Fargate (CloudFormation) template files in the Downloads section at the bottom of this article.
  2. Log in to your AWS account and navigate to the CloudFormation service.
  3. Choose Create Stack.
  4. Select Template is ready.
  5. Select Upload a template file and then Choose File.
  6. Upload your template .json and click Next to move to the next screen.

Using the quick-create Link

  1. Click on the quick-create link, which will launch the CloudFormation stack in the AWS Console with pre-filled values of some parameters.
  2. Provide the details of other parameters to create the stack and follow the Next Steps.

Specify stack details

Complete the form with the required details:

Field Description
Stack name An arbitrary name given to the created stack.
AgentID ID_AGENT attained during agent creation.
AssignPublicIp ENABLED if your subnet running the CDC Agent uses internet gateway for internet access. DISABLED if your subnet uses a internet gateway for routing traffic to the internet. Note that you cannot deploy the CDC agent on a subnet that doesn't have an internet gateway or a NAT gateway/instance. Please refer to Subnets and Security Groups for more information.
BucketName An arbitrary name for the new target bucket for CDC output. Must be unique.
ClusterName An arbitrary name for the new ECS Fargate cluster that will be created from the template. This is where your agent is hosted. Must be unique.
ExecutionRoleName An arbitrary name for the new ECS Task Execution Role that will be created. Must be unique.
ImageURL The path to the agent image on ECR. Do not edit this field.
LogGroupName An arbitrary name for the CloudWatch Log Group that will be created for agent logging. Must be unique.
OrganizationID ID_ORGANIZATION attained during agent creation.
PlatformKeyName The name of the secret generated to hold your Platform Key. If you are following our recommended install, this will be agent-rsa.
PlatformWebsocketEndpoint This is a parameter that needs to have its value set. If you are following our recommended install of the CDC agent, this will be platform-websocket-endpoint.
Region The name of the AWS Region you want to create these resources in. Note that it's usually best to keep all your AWS resources in the same region when possible.
RoleName An arbitrary name for the new IAM Task Role used by the image to access AWS resources.
SecretARNs A comma-separated list of secret ARNs. The agent needs access to the Platform Key secret as well as database password secrets. This field can be left as * to indicate access to all Secret ARNs.
SecurityGroups Select the Security group.
ServiceName An arbitrary name for the new Elastic Container Service to run tasks under. Must be unique.
SubnetIDs The Subnet ID for subnet your chosen subnet.
TaskDefinitionName A unique, arbitrary name for your agent tasks.

Next steps:

  1. When your details are entered, click Next.
  2. Configure Stack Options: No action is required on this page. Click Next.
  3. Review: Select Acknowledge the template has changed and click Create.

When your stack has been fully created (this may take a few minutes) you can return to Matillion CDC and view the Agents list. Your new agent should appear on the list and have the Connected status if all has gone well. If not, please consult our troubleshooting documentation.

If you are creating a pipeline and have completed this step, consult the CDC Pipelines documentation to review the process and find your next steps.

Downloads

This article covers the advanced template installation manually by downloading the template below.

And an accompanying parameter file for those installing via CLI:

What's Next

Cookie consent

We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content based on your browsing and navigation history.