Guides

Matillion Services

2 Articles in this category

AWS Services

5 Articles in this category

Azure Services

2 Articles in this category

GCP Services

1 Articles in this category

API Profiles

12 Articles in this category

Common Operations

11 Articles in this category

Installation Configuration

3 Articles in this category

Triggering Matillion ETL

5 Articles in this category

Collibra Integration

Introduction Collibra provides data lineage from the SQL generated from METL. The integration works by way of running a harvester on the customer environment which maps out lineage information from Matillion ETL jobs and the data sources you...

Updated on : 05 May 2022

Connecting to an RDS in a Private VPC

Overview There are following scenarios for accessing an RDS DB instance by the Matillion Instance. A DB Instance in a VPC Accessed by Matillion Instance in the Same VPC. Same VPC Important Information The simplest way to man...

Updated on : 05 May 2022

Diagnostic Data Policy

Overview This article describe's Matillion's policy and procedures concerning the handling of diagnostic data. Diagnostic data is any data provided by a customer to the Matillion support team in order to assist in identifying and resolving a c...

Updated on : 05 May 2022

Executing Python Scripts Outside of Matillion

Overview Matillion has the functionality to run Python scripts as part of a Matillion job run. This is used for setting variable values, running quick queries on the target database using database cursors, or connecting to other AWS servic...

Updated on : 05 May 2022

Helping with the GDPR

Overview From 25/5/2018 any company operating in the EU will be required to comply with the new GDPR Data Protection legislation if they store or process personal information relating to an individual. If you’re a Matillion ETL user your i...

Updated on : 05 May 2022

Integrating Slack with Matillion ETL

Overview Slack is a popular messaging tool for teams to communicate and to automate notifications from third-party applications. Matillion ETL can send messages to Slack via Amazon SNS (Simple Notification Service) messages. This can be used...

Updated on : 05 May 2022

Making Multiple API Queries

Overview Matillion ETL's API Query component is ideal for pulling data from a REST or SOAP API. However, there are cases where using a Python script to make the API call may be more efficient. As an example, suppose you have a list of addres...

Updated on : 05 May 2022

Matillion Data Quality Framework

Overview The term “data quality” is very broad, and typically has a dual meaning: Data that is complete, accurate and relevant A set of processes to achieve such a state We’ll focus on the first of these two angles here, a...

Updated on : 05 May 2022

NRT Replication In Redshift

Overview Amazon Redshift offers a fast and scalable data processing platform at around 1/10th the cost of a typical equivalent on-premises solution. This makes it perfect for integrating multiple different source systems, to produce timely, ...

Updated on : 05 May 2022

Pivoting and Unpivoting Tables

Pivoting Data System databases can often hold data in a highly normalised form such as the below: However this format is not so user friendly and is not ideal for an end user BI tool. The Transpose Rows Component can be used with the Spl...

Updated on : 05 May 2022

SCM Integration

Overview For many customers, best practice topology requires multiple instances of Matillion ETL (for development, testing and live) combined with a Source Control Management (SCM) system combined and build server or script environment to manage...

Updated on : 05 May 2022

Tracking Loaded Files

Tracking Loaded Files A common use case for Matillion ETL is to use Matillion’s SQS integration to trigger the loading of a file from S3 into Redshift using lambda. Further details of this are here . Suppose some transformations should be ...

Updated on : 05 May 2022

Turning a Backup of Matillion ETL into a new pre-release Instance

These instructions explain how to take a backup of an existing Matillion instance and then switch the new instance to the pre-release repository. This will then allow you to upgrade the instance to the pre-release version. Pre-release version...

Updated on : 05 May 2022

Using Grid Variables to Apply Business Rules in a Transformation Job

Overview When Transforming data, it is often the case that business rules need to be applied to data to do different calculations on different rows of data depending on certain criteria. Traditionally these business rules would be hard coded...

Updated on : 05 May 2022

Using incron to automatically copy data to S3

Overview This is not strictly related to Matillion ETL however it is a common task to get data easily into S3 from a premise or system. This approach is especially used in a micro-batching scenario . The below instructions should be enou...

Updated on : 05 May 2022

Using KMS encrypted passwords in Python

Overview Typical uses of Python scripts in Matillion ETL often involve the inclusion of credentials. It is always ill-advised to store a password in plaintext in a component. Unfortunately, Matillion ETL does not allow Python scripts to ta...

Updated on : 05 May 2022

Using R with Matillion ETL for Redshift

Overview Matillion ETL for Redshift runs as an EC2 Linux instance in your AWS VPC. You have full administrative access to the EC2 instance, and can install additional software - such as the statistical package R . From within Matillion yo...

Updated on : 05 May 2022

Using Table Metadata to Grid

Overview The Table Metadata To Grid component in Matillion ETL takes metadata from a table and uses this metadata to populate a grid variable . This page offers an example of how and why the component might be used. In this example we...

Updated on : 05 May 2022

Managing Python on a Matillion ETL Virtual Machine (VM)

Overview Note This guide was provided by Matillion's solution architect team. Python is a popular programming language for data engineers. Recognizing this, Matillion ETL offers a Python Script component, which allows Matillion users t...

Updated on : 05 May 2022