Getting Started

Getting started

To get started with Matillion Data Loader, you will need:

An active Matillion Hub account. For more information, read Matillion Hub Overview.
A source application or database where your data resides.
A destination where you want to load the data to.
Credentials to allow Matillion Data Loader to connect with the source and destination systems.

Every CDC pipeline requires a Matillion Data Loader CDC Agent to orchestrate the data loading tasks.

Agents must be installed within the cloud service provider of your choice before a CDC pipeline can be created.
The CDC Agent requires access to the source database, the target cloud data platform, and a secrets management application, either AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. A platform key is generated (if not already done for this account) and stored in AWS Secrets Manager, Azure Key Vault, or Google Secret Manager.
Cloud resources are created for use by the Agent such as storage and logs. See specific installation guides for more information:
Amazon S3.
Azure Blob.
Google Cloud Storage.
The Agent must be deployed to the appropriate environment with the correct environment variables.
Agent status must be Connected for a CDC pipeline to succeed.
Once the CDC Agent is connected, you can create CDC pipelines.
Please refer to agent installation documentation for detailed information and consult your cloud administrator for help and permissions where required.

A source application or database where your data resides.
A database or data warehouse destination to which the data must be replicated.
Access for Matillion Data Loader to connect with the source and the destination systems.

A cloud data warehouse or cloud storage, to store your data. This can also be the destination warehouse that holds the data from your various CDC/Batch pipelines.
The data warehouse can be an existing destination or an external repository that you've set up for your pipelines. To use an existing destination as the active data warehouse, you must offer Matillion extra permissions.
Matillion currently supports the following cloud data warehouses:
Snowflake.
Amazon Redshift.
Google BigQuery.
Amazon S3.
Azure Blob Storage.
Google Cloud Storage.
Access for Matillion Data Loader to connect to the warehouse systems.

To understand the technical requirement for data warehouse setup, please read Technical Requirements.

For more information about signing up for Matillion Data Loader, please read Signing up for Matillion Data Loader.

For a general understanding of CDC data loading, read CDC Overview.
For an understanding of batch data loading, read Batch Loading Overview.
For a more detailed look at the features and user interface of Matillion Data Loader, read MDL Dashboard Overview.

Table of contents