Loading Data

Loading Data


Overview

'Loading' data in Matillion means taking a data source and converting into a table on a target platform. Matillion ETL offers many ways to load data through its components that can source data from many different partners.

 

Loading Data Features

Each of the components linked below allow data loading/unloading to and from platform-specific services.

AWS:

GCP:

 

Query Components

The Data Staging 'Query' Components allow Matillion ETL to query data from various external sources and load the data into a table ready for transformation. This process usually involves staging the data on a file storage area depending on your platform.

A list of Query Components and their data models, OAuth guides and related articles can be found here.


Load Status

Currently-loading data on the Matillion instance can be viewed through Project -> Load Status. This window allows a user to see all current loads, regardless of the job or user they belong to, in a single window. When loading stops (be it via completion, cancellation or failure) the load is removed from Load Status.

Each load is detailed with its ID, Start Time, (current) Duration, (current) Rows and (current) Progress. A load may be selected and expanded to reveal further information on the data.

The window may be refreshed at will using the 'Refresh' button.


Incremental Loading

It is often the case where a single source is regularly updated with new data and it is useful for Matillion ETL to load only the new data and use it to update other tables. For explanation and examples of how to incrementally load data using Matillion ETL, please refer to the Incremental Load Documentation.