Launching Matillion ETL using CloudFormation Templates
This guide details parameters for creating a stack in AWS using CloudFormation templates. These steps are important for users wanting to launch a Matillion ETL instance on AWS via Cloudformation Templates, regardless of the chosen cloud data warehouse.
- This document is part of a series on Matillion ETL Instance Creation.
- Users responsible for launching from the Matillion Hub, require valid permissions to create and make changes in the AWS and ought to know how to create stack using CloudFormation template.
- Before you create a stack from a CloudFormation template in AWS, you must ensure that all dependent parameters that the template requires are already defined.
- See our List of CloudFormation Templates for links to the templates.
In the process of creating Matillion ETL instance through the Matillion Hub, You will select AWS (Cloud Provider), Choose a Cloud Data Platform, CloudFormation Template (Delivery Method), and follow the selections on the screen. For more general information on launching from the Matillion Hub and the options available, please check Matillion ETL Instance Creation.
Based on your selection, the system will pre-populate the CloudFormation template for you to review in AWS. Please make sure you have required permissions to make necessary changes in AWS.
After clicking Continue in AWS you will be redirected to AWS services to Quick Create Stack page followed by Cloud Formation, Stacks, then Create stack.
If you use the CloudFormation template to create a Matillion ETL instance, the parameters which are defined in your template will be pre-populated and allow you to input custom values. In our case, you will be redirected to the AWS services stack page with pre-populated stack template, stack name and, values for the parameters that were defined already in the CloudFormation template.
- A template can use or refer to both existing AWS parameters and parameters declared in the template itself. AWS CloudFormation takes care of checking references to parameters in the template and also checks references to existing parameters to ensure that they exist in the region where you are
creating the stack.
- Although you may see some references to Redshift in the screenshots, this process is applicable regardless of the chosen Cloud Data Platform and any references are just for the sake of example, here.
Specifying stack parameters
Select the EC2 Instance type using dropdown, on which to run Matillion ETL instance. It is a pre-populated value, as the CloudFormation template has defined this value and passed into the parameter. For more information on using Instance type in AWS, you can visit Instance Sizes in the Matillion Hub documentation. Before launching Matillion ETL instance, it is worth understanding the available instance types and their limits, allowing you to choose the appropriate instance type for your team.
Network and Security Configuration
The CloudFormation template contains an input parameter, KeyName, that specifies the key pair used for the Amazon EC2 instance that is declared in the template. The template depends on the user who creates a stack from the template to supply a valid Amazon EC2 key pair for the KeyName parameter. If you supply a valid key pair name, the stack creates successfully. If you don't supply a valid key pair name, the stack is rolled back.
Make sure you have a valid Amazon EC2 key pair and record the key pair name before you create the stack. To see your key pairs, open the Amazon EC2 console, then click Key Pairs in the navigation pane.
If you don't have an Amazon EC2 key pair, you must create the key pair in the same region where you are creating the stack. For information about creating a key pair, see Create a key pair using Amazon EC2.
VPC IPv4 CIDR
A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your AWS resources, such as Amazon EC2 instances, into your VPC.
When you create a VPC, you must specify a range of IPv4 addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. You cannot specify an IPv4 CIDR block larger than /16. This is the primary CIDR block for your VPC. VPC IPv4 CIDR is a pre-populated field as it is been already configured into the CloudFormation template and passed value into the fieldset.
This allows you to select an Availability Zone from the current region using the drop-down menu.
Ensure that Public Subnet CIDR (Must be within VPC CIDR range). It will be a pre-populated field as its already been configured while creating VPC.
Inbound IPv4 CIDR
Inbound IPv4 CIDR Range for Matillion Instance is 0.0.0.0/0
If you have chosen to use Amazon Redshift as your Cloud Data Platform, it will be necessary to configure a Redshift Cluster as part of your Stack. To complete this, refer to the Redshift Configuration for Matillion ETL.
Once, you finish with all the details required in the parameters, you click Create Stack. After the successful creation of the instance you can launch your Matillion ETL instance.
See the following documentation for further guidance where necessary