Launching Matillion ETL using Amazon Machine Image
This is a guide to providing a detailed description on specifying parameters required in launching Matillion ETL using Amazon Machine Image(AMI). As an example, in some steps throughout this guide, you will notice that we use Delta Lake images as the Cloud Data Platform. These steps apply to all Cloud Data Platforms.
- This document is part of the Matillion ETL Instance Creation process.
- Users responsible for launching from the Matillion Hub, require valid permissions to create and make changes in the AWS and ought to know how to create stack using Amazon Machine Image on Amazon EC2 or build a private AMI.
To launch and configure Matillion ETL using Amazon Machine Image, you should select AMI from the Billing setup process and start the launch/setup procedure. To do this, use the following steps:
- Log into Matillion Hub. You will be required to sign in using your own credentials. Choose your organization, click Add Matillion ETL instance on the Select your service page to begin the process of creating an instance. Then make sure you choose AWS as your cloud provider.
- Next, select your preferred Cloud Data Platform from the available options.
- On the next page, select Amazon Machine Image(AMI) as an option to deliver Matillion ETL.
- Please follow the instructions further on the screen as discussed in Matillion ETL Instance Creation via the Matillion Hub documentation. Later, click Continue in AWS as shown in the image below.
- Based on your selection, you will be redirected to the AWS account to launch an instance. The system will pre-populate the Amazon Machine Image (AMI) with all pre-configured details required to launch the instance. Click Launch situated at the top-left of the page.
If choosing from a list of AMIs, use only AMIs with "billing" in the instance name and not "byol". Also note that images are not necessarily listed in order of recency and care should be taken when selecting the desired version.
- The browser will redirect you to the Choose an instance type page. Choose one of the supported Instance Types, and click Next: Configure Instance Details.
- This is a manual setup instead of a 1-click setup. There are some settings you can specify on this screen that are unavailable through 1-click, that improve the functionality of your Matillion Instance. Please follow the configurations as shown below:
- Number of instances: leave as default (unless you want multiple instances).
- Purchasing options: leave as default i.e. unchecked.
- Network: Choose a VPC. The zone which this VPC is located should ideally be the same as your Snowflake account (European Snowflake accounts should be paired with a EU-based AWS region, for example).
- Subnet: Choose a subnet. Each subnet resides in one availability zone.
- Auto-assign public IP: This depends very much on the setup, by default a new VPC will not have VPN connections or NAT Gateways available, so in order for Matillion ETL to connect to the internet and for you to access Matillion this will normally need to be set to Enable.
- IAM Role: IAM Roles are used to allow your Instances to use the Amazon API's securely without manual management of security keys. To use all the features of Matillion ETL, you should configure an IAM Role for your instance to use. This procedure assumes you do not already have an appropriate IAM role setup (if you do, simply select it). For detailed information on creating IAM roles, read IAM Roles & Permissions (AWS).
- Choose below settings for Shutdown behaviour, Enable termination protection, Monitoring, and Tenancy:
- Shutdown behaviour: leave as default i.e. Stop.
- Enable termination protection: we suggest setting to, ‘Enabled’.
- Monitoring: we suggest setting to, ‘Enabled’.
- Tenancy: Run a shared hardware instance.
Once you have completed all the details, click Next - Add Storage.
You could Review and Launch at this step, if you do not want to add any storage, tags or configure security groups. The system will redirect you to the Review page.
- On the Add Storage page, you can choose the default root volume size and click Next - Tag Instance.
- Next, at the Add Tags page, add any instance tags your require. For example, you could define a tag with key = Name and value = Webserver. A copy of a tag can be applied to volumes, instances or both. Then, click Next – Configure Security Group.
- On the Configure Security Group page, a default security group will be created with the minimum set of ports. Provide a security group name and adjust the settings of your security requirements, if needed. Next, click Review and Launch.
The default recommended security group uses SSH (port 22) and HTTP(S) (port 80 and 443) access to the instance. The range of allowed IPs should be tailored to your needs.
- Lastly, you need to review all the settings you made and configuration details for Matillion ETL AMI on the Review and Launch page, and then click Launch.
- A pop-up dialog for key pair information will appear. Select an existing key pair, or create a new one. The selected key pair will be added to the set of keys authorized for this instance. Tick the checkbox to acknowledge the information. Click Launch Instances.
- Your instances are launching, and it may take a few minutes until they are in the running state.
- Click View Instances to monitor your instances' status. Once your instances are in the running state, you can connect to them from the Instances page.
- Log into your copy of Matillion with ec2-user, and the instance ID i-xxxxxxxx (e.g. i-88ed92c6).