Launching Matillion ETL for Snowflake - AWS
Matillion ETL is an AMI-based ETL/ELT tool built specifically for Snowflake. Modern, browser-based UI. Powerful, push-down ETL/ELT. This page describes how to launch and connect to Matillion ETL from the AWS Marketplace.
If you have already completed these steps and wish to connect Matillion ETL to Snowflake, please refer to the documentation here.
This article describes how to launch and connect to Matillion ETL for Snowflake from the AWS Marketplace.
Launching Matillion ETL for Snowflake
Launching Matillion ETL from the AWS Marketplace
To launch and configure Matillion ETL for Snowflake, you should select it from the AWS Marketplace and start the launch/setup procedure. To do this, use the following steps:
- Locate Matillion ETL for Snowflake on the AWS Marketplace.
- From the Matillion ETL for Snowflake product page on the AWS Marketplace, press the Continue to Subscribe button on the top right of the Subscribe to this software page.
- Depending on your current setup, you may also need to click Continue to Configuration on the new page.
- On the Configure this software page, select Amazon Machine Image with a 64-bit Amazon Machine Image (AMI) as your Fulfillment Option and then, click Continue to Launch to further process the steps.
It is highly recommended that users always select the most up-to-date product version available and in a region in which they are selecting to host their instance.
Launching the ETL at this stage will not launch your instance or will not charge to your account.
Launch this software
- You will be redirected to the Launch this software page. To launch the software from the website, simply select Launch through EC2 using the dropdown. And, click Launch.
- The browser will redirect you to the Choose an instance type page. Choose one of the supported Instance Types and click Next: Configure Instance Details
- t3.medium: For the teams of 1-2 data professionals using Matillion ETL concurrently.
- m5.large: For the teams of 3-5 data professionals using Matillion ETL concurrently.
- r5 type instance: For the production instances that run jobs but need few users.
- This is a manual setup instead of a 1-click setup. There are some settings you can specify on this screen that are unavailable through 1-click, that improve the functionality of your Matillion Instance. Please follow the configurations as shown below:
If you are not already logged into AWS Marketpace, you will be prompted for your AWS account credentials.
Choosing the Instance Type
Each instance type is appropriately sized to support a given number of users and the software itself recognises the type of instance it is running on and restricts maximum concurrent users on this basis.
The instance type you select affects the number of users that can use Matillion ETL concurrently. For more information see here.
You can read more about Instance Types and pricing at here.
Configure Instance Details
- Number of instances: leave as default (unless you want multiple instances).
- Purchasing options: leave as default i.e. unchecked.
- Network: Choose a VPC. The zone which this VPC is located should ideally be the same as your Snowflake account (European Snowflake accounts should be paired with a EU-based AWS region, for example).
- Subnet: Choose a subnet. Each subnet resides in one availability zone.
- Auto-assign public IP: This depends very much on the setup, by default a new VPC will not have VPN connections or NAT Gateways available, so in order for Matillion ETL to connect to the internet and for you to access Matillion this will normally need to be set to Enable.
- IAM Role: IAM Roles are used to allow your Instances to use the Amazon API's securely without manual management of security keys. To use all the features of Matillion ETL, you should configure an IAM Role for your instance to use. This procedure assumes you do not already have an appropriate IAM role setup (if you do, simply select it). For detail information on creating IAM roles, you can visit Additional Information section of this guide.
- Choose below settings for Shutdown behaviour, Enable termination protection, Monitoring, and Tenancy:
- Shutdown behaviour: leave as default i.e. Stop.
- Enable termination protection: we suggest setting to, ‘Enabled’.
- Monitoring: we suggest setting to, ‘Enabled’.
- Tenancy: Run a shared hardware instance.
Once you have completed all the details, click Next - Add Storage.
You can attach additional EBS volumes and instance store volumes to your instance, or edit the settings of the root volume. You can also attach additional EBS volumes after launching an instance, but not instance store volumes. We recommend increasing volumen by the factor of 5.
The default recommended security group uses SSH (port 22) and HTTP(S) (port 80 and 443) access to the instance. The range of allowed IPs should be tailored to your needs.
Once the AMI has initialised, which normally takes a few minutes, you can access Matillion ETL by entering the hostname of the instance into a web browser.
Log in to your copy of Matillion with ec2-user and the instance ID i-xxxxxxxx (e.g. i-88ed92c6)
This section will guide you to "create a new IAM role" while configuring instance details in the AWS Console.
Create New IAM Role
- Once you are on Configure Instance Detail page, click Create New IAM Role to create a new IAM Role, .
- The browser will redirect you to the AWS Console → Identity and Access Management (IAM) → Roles, click Create Role.
- Select the type of trusted entity on the next page (Preferrable is: AWS Service), and choose a common use case from the available list (Preferred use case is : EC2). Once done, click Next:Permission.
- Next, you need to attach a Permission Policy. You can choose from the existing policy or can create a new policy if needed. Then, click Next:Tags.
From the list select:
- Add IAM Tags. IAM tags are key-value pairs you can add to your role. Tags can include user information, such as an email address, or can be descriptive, such as a job title. Click Next:Review .
- On the Review page, provide the required information and review this role before you create it and click Create Role.
- Finally, the IAM role has been created and you will find the newly created role in the list.
For more information on the policies see Managing Credentials.