Launching Matillion ETL for Snowflake - AWS

Launching Matillion ETL for Snowflake - AWS


Overview

Matillion ETL is an AMI-based ETL/ELT tool built specifically for Snowflake. Modern, browser-based UI. Powerful, push-down ETL/ELT. This page describes how to launch and connect to Matillion ETL from the AWS Marketplace.

If you have already completed these steps and wish to connect Matillion ETL to Snowflake, please refer to the documentation here.

This article describes how to launch and connect to Matillion ETL for Snowflake from the AWS Marketplace.



Launching Matillion ETL for Snowflake

Launching Matillion ETL from the AWS Marketplace

To launch and configure Matillion ETL for Snowflake, you should select it from the AWS Marketplace and start the launch/setup procedure. To do this, use the following steps:

  1. Locate Matillion ETL for Snowflake on the AWS Marketplace.
  2. Locate Matillin ETL in AWS Marketplace

    Locate Matillion ETL in AWS Marketplace

  3. From the Matillion ETL for Snowflake product page on the AWS Marketplace, press the Continue to Subscribe button on the top right of the Subscribe to this software page.
  4. Continue to Subscribe

    Continue to Subscribe

    Please Note

    If you are not already logged into AWS Marketpace, you will be prompted for your AWS account credentials.

  5. Depending on your current setup, you may also need to click Continue to Configuration on the new page.
  6. Continue to Configuration

    Continue to Configuration

  7. On the Configure this software page, select Amazon Machine Image with a  64-bit Amazon Machine Image (AMI) as your Fulfillment Option and then, click Continue to Launch to further process the steps. Select Amazon Machine Image

    Select Amazon Machine Image

    Please Note

    It is highly recommended that users always select the most up-to-date product version available and in a region in which they are selecting to host their instance.


    Launching the ETL at this stage will not launch your instance or will not charge to your account.

    Launch this software

  8. You will be redirected to the Launch this software page. To launch the software from the website, simply select Launch through EC2 using the dropdown. And, click Launch.
  9. Launch from Website

    Launch from Website

    Choosing the Instance Type

  10. The browser will redirect you to the Choose an instance type page. Choose one of the supported Instance Types and click Next: Configure Instance Details
  11. Instancne Type

    Instancne Type

    Important Information

    Each instance type is appropriately sized to support a given number of users and the software itself recognises the type of instance it is running on and restricts maximum concurrent users on this basis.


    The instance type you select affects the number of users that can use Matillion ETL concurrently. For more information see here.

    • t3.medium: For the teams of 1-2 data professionals using Matillion ETL concurrently.
    • m5.large: For the teams of 3-5 data professionals using Matillion ETL concurrently.
    • r5 type instance: For the production instances that run jobs but need few users.

    You can read more about Instance Types and pricing at here.

    Configure Instance Details

  12. This is a manual setup instead of a 1-click setup. There are some settings you can specify on this screen that are unavailable through 1-click, that improve the functionality of your Matillion Instance. Please follow the configurations as shown below:
    • Number of instances: leave as default (unless you want multiple instances).
    • Purchasing options: leave as default i.e. unchecked.
    • Network: Choose a VPC. The zone which this VPC is located should ideally be the same as your Snowflake account (European Snowflake accounts should be paired with a EU-based AWS region, for example).
    • Subnet: Choose a subnet. Each subnet resides in one availability zone.
    • Auto-assign public IP:  This depends very much on the setup, by default a new VPC will not have VPN connections or NAT Gateways available, so in order for Matillion ETL to connect to the internet and for you to access Matillion this will normally need to be set to Enable.
    • IAM Role: IAM Roles are used to allow your Instances to use the Amazon API's securely without manual management of security keys. To use all the features of Matillion ETL, you should configure an IAM Role for your instance to use. This procedure assumes you do not already have an appropriate IAM role setup (if you do, simply select it). For detail information on creating IAM roles, you can visit Additional Information section of this guide.
    • Choose below settings for Shutdown behaviour, Enable termination protection, Monitoring, and Tenancy:
      • Shutdown behaviour: leave as default i.e. Stop.
      • Enable termination protection: we suggest setting to, ‘Enabled’.
      • Monitoring: we suggest setting to, ‘Enabled’.
      • Tenancy: Run a shared hardware instance.
    Configuration Details

    Configuration Details

    Once you have completed all the details, click Next - Add Storage.

  13. On the Add Storage page, you can choose the default root volume size and click Next - Tag Instance.
  14. Add Storage

    Add Storage

    Please Note

    You can attach additional EBS volumes and instance store volumes to your instance, or edit the settings of the root volume. You can also attach additional EBS volumes after launching an instance, but not instance store volumes. We recommend increasing volumen by the factor of 5.

  15. Next, at the Add Tags page, add any instance tags your require. For example, you could define a tag with key = Name and value = Webserver. A copy of a tag can be applied to volumes, instances or both. Then, click Next – Configure Security Group
  16. Add Tags

    Add Tags

  17. On the Configure Security Group page, A default security group will be created with the minimum set of ports. If required, name and adjust as to your security requirements. Next, click Review and Launch
  18. Security Group

    Security Group

    Please Note

    The default recommended security group uses SSH (port 22) and HTTP(S) (port 80 and 443) access to the instance. The range of allowed IPs should be tailored to your needs.

  19. Lastly, you need to review all the settings you made and configuration details for Matillion ETL for Snowflake AMI on Review and Launch page and then, click Launch
  20. Review and Launch

    Review and Launch

    Important Information

    Once the AMI has initialised, which normally takes a few minutes, you can access Matillion ETL by entering the hostname of the instance into a web browser.

    Log in to your copy of Matillion with ec2-user and the instance ID i-xxxxxxxx (e.g. i-88ed92c6)


Additional Information

This section will guide you to "create a new IAM role" while configuring instance details in the AWS Console.

Create New IAM Role

  1. Once you are on Configure Instance Detail page, click Create New IAM Role to create a new IAM Role, .
  2. Configure Instance detail

    Configure Instance detail

  3. The browser will redirect you to the AWS ConsoleIdentity and Access Management (IAM)Roles, click Create Role.
  4. AWS Console - IAM

    AWS Console - IAM

  5. Select the type of trusted entity on the next page (Preferrable is: AWS Service), and choose a common use case from the available list (Preferred use case is : EC2). Once done, click Next:Permission.
  6. Trusted entity and User Case

    Trusted entity and User Case

  7. Next, you need to attach a Permission Policy. You can choose from the existing policy or can create a new policy if needed. Then, click Next:Tags.
    From the list select:
    • AmazonSNSFullAccess
    • AmazonSQSFullAccess
    • CloudwatchFullAccess
    • AmazonRDSReadOnlyAccess
    • AmazonS3FullAccess
  8. Permission Policy

    Permission Policy

    Please Note

    For more information on the policies see Managing Credentials.

  9. Add IAM Tags. IAM tags are key-value pairs you can add to your role. Tags can include user information, such as an email address, or can be descriptive, such as a job title. Click Next:Review .
  10. IAM Tags

    IAM Tags

  11. On the Review page, provide the required information and review this role before you create it and click Create Role.
  12. Review IAM Role

    Review IAM Role

  13. Finally, the IAM role has been created and you will find the newly created role in the list.
  14. New IAM Role Created

    New IAM Role Created



Next Steps