ETL agents overview

Welcome to the documentation for Matillion ETL Agent

The Matillion ETL Agent allows a compatible Matillion ETL job to be executed and scheduled via the Matillion Platform and in a user's data plane by use of an ETL Agent—benefiting from enhanced scalability capabilities of this new architecture. The data plane and control plane split ensures data sovereignty is adhered to.

What's a data plane?

This is an abstract concept that ties together an execution environment—for example, AWS Fargate—that uses an ETL Agent. This ETL Agent will be provided by Matillion as a Docker image.

What's the ETL Agent?

The ETL Agent is the software that executes the job tasks that originally ran within Matillion ETL. The ETL Agent communicates with the Agent Gateway by an egress-only method. This provides the job tasks—from the SaaS platform—that will be executed. The ETL Agent can scale up by having many instances running concurrently and this can be set based on requirements. There are plans to make this scalable; however, at this moment, they are sized manually.

What's the Agent Gateway?

The Agent Gateway is the communication portal that agents will use to communicate with the SaaS Platform. The ETL Agents poll the Agent Gateway for work, as well as send responses for those work tasks.

What is the ETL Agent Manager?

This is the application where you create your configuration used for installing a new ETL agent. It is accessed from the Matillion Hub by selecting Manage ETL Agents. It also can facilitate the automatic updating of agents when installed in a compatible environment such as AWS ECS Fargate.

Considerations

• To run ETL jobs using Unlimited Scale or the Matillion Platform in your own virtual cloud provider, you must create an ETL Agent. For this to work successfully, you will need to create a number of resources in Amazon Web Services (AWS).
• Matillion recommends that credentials are stored externally where possible—for example, AWS Secrets Manager.
• OAuth entries may need to be recreated—such as for Salesforce Query.

Matillion doesn't recommend:

• Running production workloads during the private preview.
• Running workloads with sensitive data during the private preview period.

Help us improve

The Matillion ETL Agent is still in development. Please get in touch with any problems you encounter while using this product.