This article is intended for customers who are registered with the Matillion Hub. Follow the instance sizes recommendations for launching a Matillion ETL instance through the Matillion Hub.
This document explains the factors that affect the performance of your Matillion ETL instance. There are three important factors that determine Matillion ETL instance performance:
- CPU: The number of virtual CPU cores.
- Memory: The available memory to process instance data.
- Network: The quality, speed, and capacity of your network connection.
Both your edition and the physical requirements of your Matillion ETL instance, as well as its users, are key factors to consider when choosing your instance size. We will discuss these factors in detail to help you make the best decision for your needs.
Matillion ETL is tested to run on instances with processors using x86_64 instruction sets and should be used as such. Other instance types—such as those using ARM processors—should be avoided and in many cases can't be selected for use with Matillion ETL.
The number of vCPU cores is directly related to the maximum performance of your Matillion ETL instance. Broadly speaking, more cores result in better performance and insufficient cores could result in computational delays and even fatal processing errors, particularly under heavy scheduling or CDC. Different editions provide more features which will in turn require greater CPU usage.
The number of vCPUs dictates the number of tasks an instance can execute at one time. Matillion ETL can run 16 jobs at a time, and each job can run 2 parallel tasks per vCPU. The total number of concurrent tasks per vCPU is 32 (16 Jobs × 2 Tasks × vCPU). For example, a 2 vCPU machine can handle a total of 64 parallel tasks. For more information, read Job Concurrency.
Instances with more vCPUs consume more credits per hour. For more information, read Credit Consumption Terms. As such, it's important that you choose the appropriate CPU for your needs.
Insufficient memory will cause your Matillion ETL instance to fail. The required memory directly scales with the number of tasks being processed and the number of concurrent users. It's recommended that you increase your memory if you want to have more concurrent users.
Please contact firstname.lastname@example.org if you require further help and advice when choosing the correct Matillion ETL instance size for your needs.
The size of your Matillion ETL instance has a direct effect on your credit consumption in the Matillion Hub. Matillion ETL instances with more vCPUs consume more credits per hour, and credit consumption is per vCPU per hour. Please choose carefully when considering your edition and making your instance selections. Read Credit Consumption Dashboard for more information.
- For information about Credits in the Matillion Hub, and how they're consumed, read 2. Matillion ETL Instances Usage in Credit Consumption Terms.
- The cost of your AWS or Azure payment subscription is not included in your Matillion Hub billing. Read Credit Consumption Dashboard for more information.
Matillion ETL will run on many different instance sizes providing they use an x86_64 instruction set. The following tables give just a few example instance sizes for AWS and Azure. Please read, Amazon EC2 and Azure Virtual Machines for more specific information.
When making your instance size selection, we recommend starting with the minimum CPU and memory that satisfies your requirements, and increase as needed.
The minimum Matillion ETL instance size for AWS and Azure is: 2vCPU and 4GiB of memory. GCP is not currently supported in the Matillion Hub.
More vCPUs means higher credit consumption. Additionally, the cloud cost of your VM is not included in credit pricing and generally cost more to run with higher specifications.
|Name||vCPU||Concurrent Tasks||Memory (GiB)||Notes|
|m5.large||2||64||8||A balanced starter instance with relatively low AWS costs and credit consumption.|
|m5.xlarge||4||128||16||A larger instance leaning toward heavy production use. At 4 vCPU, it can handle double the tasks of an m4.large with 2vCPU and will consume credits twice as quickly.|
|m5.2xlarge||8||256||32||An extreme production instance for heavy workloads and high concurrent user count. Consumes 8 credits per active hour, 4 × more than the m4.large.|
|Name||vCPU||Concurrent Tasks||Memory (GiB)||Notes|
|B2s||2||64||4||A balanced starter instance with relatively low Azure costs and credit consumption.|
|D4s_v4||4||128||16||A larger instance leaning toward heavy production use. At 4 vCPU, it can handle double the tasks of B2s with 2vCPU and will consume credits twice as quickly.|
|D8s_v4||8||256||32||An extreme production instance for heavy workloads and high concurrent user count. Consumes 8 credits per active hour, 4 × more than the B2s.|
Choosing your instance size
Use the following steps to choose a suitable instance size for your Matillion ETL instance:
- Access and sign in to the Matillion Hub. If you are a new customer, you must register for a Matillion Hub account before you can launch a Matillion ETL instance.
- Follow the steps for creating a Matillion ETL instance.
- Once you have been re-directed to your cloud provider's console page, you will be required to configure your new Matillion ETL instance. Follow the steps listed in your chosen console:
- Refer to AWS instance recommendations or Azure instance recommendations for recommendations on the instance size that best suits your needs. For more information, read Amazon Documentation on instance sizes, or Azure Documentation on instance sizes, respectively.
- Review your configuration settings in the cloud provider console, and create your Matillion ETL instance.