CloudFormation EC2 ECS Proxy Templates
This document outlines the prerequisites and steps for the deployment of the Matillion CDC agent on EC2 ECS and behind a web proxy server.
- An active Amazon Web Services (AWS) account with the requisite permissions to add inbound rules and to set network and security settings.
- Access to AWS VPC.
- Two or more private subnets within the target VPC that require access to a NAT gateway.
- One public subnet within a NAT gateway and routing tables configured.
- URLs for HTTP and HTTPS proxy server.
Please contact your cloud network engineer for the required permissions and network settings.
The list of required parameters from AWS/Matillion Data Loader:
|VpcId||AWS Cloud Portal > VPC||AWS||vpc-xxxxxxxxxx|
|PrivateSubnetId||AWS Cloud Portal > Subnets||AWS||subnet-xxxxxx|
|SecurityGroupId||AWS Cloud Portal > EC2 > Security Groups||AWS||sg-xxxxxxx|
|SSHKeyName||AWS Cloud Portal > EC2 > Key Pair||AWS||Name of key pair|
|AgentId||Matillion Hub > MDL > Agents > Add Agent (Prerequisite page)||Matillion Data Loader||111111111-2222-3333-3333-44444444444|
|OrganizationId||Matillion Hub > MDL > Agents > Add Agent(Prerequisite page)||Matillion Data Loader||111111111-2222-3333-3333-44444444444|
|PlatformKeyName||AWS Cloud Portal > Secret Manager||AWS||Certificate value stored within Secret Manager in AWS|
|PlatformWebsocketEndpoint||Matillion Hub > MDL > Agents > Add Agent(Prerequisite page)||Matillion Data Loader||
If using the proxy template, the following additional parameters are required:
|Http Proxy Url||Networking Team||Client||http://
|Https Proxy Url||Networking Team||Client||https://
If proxy requires certificate authentication:
|Certificate Download Url||Networking Team||Client||http://
|Certificate Zip Filename||Networking Team||Client||http://
Create a CDC agent in Matillion Data Loader
- Log in to the Matillion Hub.
- The My Accounts page lists any accounts you have already created or joined. At the bottom of this list, click Add new account. Read Create an Account to learn more about this topic.
Each Matillion Hub account can generate its own unique platform key that your CDC agent will use to communicate with Matillion Data Loader. With this in mind, create the CDC agent in the account that matches the platform key you will be using.
- Choose Matillion Data Loader as the service on the Select your service page.
- On the Matillion Data Loader dashboard, scroll to the lower-right of the UI and choose your region.
- Select Agents in the left sidebar and click Add agent.
- Give your agent a sensible Agent name and Description. Click Continue.
- Since this guide is for AWS, select AWS as your cloud provider.
- Choose CloudFormation as the service to provision and deploy your cloud resources from for the CDC agent installation.
- In the Prerequisites for agent setup, note the following values:
- ID_ORGANIZATION: This value is used when deploying the CDC agent in AWS. The value is unique per agent.
- ID_AGENT: Also used when deploying the CDC agent. The value is unique per agent.
- PLATFORM_WEBSOCKET_ENDPOINT: Also used when deploying the agent. The value is unique for the Matillion Data Loader region (US or EU).
- Public/Private key pair: This is a generated value. If you haven't generated a platform secret for your account yet, Matillion Data Loader will prompt you to do so when creating a CDC pipeline. You need to store this value in AWS Secrets Manager where your CDC agent can access it. For security reasons, this key pair can only be generated and shown once per account, so make sure to copy and save it for future use.
- Check the I have saved the private key in AWS Secrets Manager and made a note of the secret name checkbox.
- Click Submit key pair.
Deploy the CDC agent in AWS with a proxy
- Download the required template from the Download Templates section at the bottom of this page.
- Log in to the AWS console.
- In the AWS console, navigate to the region drop-down and select the region in which you wish to deploy the CDC agent.
Make sure the region is either
us, depending on the Matillion Data Loader region you are building the pipeline within. In the AWS console, you must choose the same region.
- Navigate to CloudFormation and click Create stack > With new resources (standard).
- Under Specify template, select Upload a template file. Select Choose file and upload the template from step 1. Click Next.
- Provide the following information for the stack details. Note that the template you choose will autofill part of the information.
- Stack Name: A unique name for the stack.
- AgentID: The value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
- Assign PublicIp: Set to Enabled.
- BucketName: The name of the S3 bucket for staging. The template should autofill this value.
- ClusterName: The name for the ECS Fargate cluster to be created for hosting your agent. The template should autofill this value.
- ExecutionRoleName: The role name for the role that ECS will use to initialize the task.
- PrivateSubnets: The subnet of the private network.
- HttpProxyUrl: URL of the HTTP proxy.
- HttpsProxyUrl: URL of the HTTPS proxy.
- ImageUrl: The URL of the repository you are pulling the CDC agent image from. The template should autofill this value.
- LogGroupName: Name of the CloudWatch Log Group for agent logging. The template should autofill this value.
- OrganizationID: This is the value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
- Owner: The owner of the resources created.
- PlatformKeyName: The name of the AWS Secrets Manager secret in which your generated private key is stored. Consult the AWS Secrets Manager documentation for more information about creating a secrets manager in AWS.
- PlatformWebSocketEndpoint: The value for
PLATFORM_WEBSOCKET_ENDPOINT, copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Matillion Data Loader.
- RoleName: The role name for the role that the task will run as. The template should autofill this value.
- SSHKeyName: The SSH key required for the server access.
- SecurityGroups: If there are any existing security groups, select them from the drop-down menu. For more detail see here
- ServiceName: The name of the ECS Service you want these tasks to run under.
- TaskDefinitionName: The name of the task definition for agent tasks.
- VpcId: Select at least two subnets in your selected VPC.
- Click Next and then Next.
- Tick the statement under Capabilities.
- Review the information you've entered, click the required check boxes and click Create Stack.
- The stack creation will then begin and should complete in approximately five minutes.
- When the stack creation is complete, the agent container will be deployed as an AWS Elastic Container Service (ECS) Cluster. The CloudFormation template also creates resources in IAM, S3, and CloudWatch Logs.
In Matillion Data Loader, your created CDC agent's status should display as Connected and offer the Add Pipeline button.
- EC2 ECS Proxy CDC Deployment Template
- EC2 ECS Proxy CDC with Certificate Deployment Template
- EC2 ECS Non-Proxy CDC Deployment Template