Contents x
Guides
19 Articles  in this category
  • Dark
    Light
Contents

Guides

19 Articles in this category
  • Dark
    Light

Matillion Services

2 Articles in this category

AWS Services

5 Articles in this category

Azure Services

1 Article in this category

GCP Services

1 Article in this category

API Profiles

12 Articles in this category

Common Operations

11 Articles in this category

Installation Configuration

3 Articles in this category

Triggering Matillion ETL

6 Articles in this category

Collibra Integration

Introduction Collibra provides data lineage from the SQL generated from METL. The integration works by way of running a harvester on the customer environment which maps out lineage information from Matillion ETL jobs and the data sources you...

Connecting to an RDS in a Private VPC

Overview There are following scenarios for accessing an RDS DB instance by the Matillion Instance. A DB Instance in a VPC Accessed by Matillion Instance in the Same VPC. Same VPC Important Information The simplest way to man...

Diagnostic Data Policy

Overview This article describe's Matillion's policy and procedures concerning the handling of diagnostic data. Diagnostic data is any data provided by a customer to the Matillion support team in order to assist in identifying and resolving a c...

Executing Python Scripts Outside of Matillion

Overview Matillion has the functionality to run Python scripts as part of a Matillion job run. This is used for setting variable values, running quick queries on the target database using database cursors, or connecting to other AWS servic...

Helping with the GDPR

Overview From 25/5/2018 any company operating in the EU will be required to comply with the new GDPR Data Protection legislation if they store or process personal information relating to an individual. If you’re a Matillion ETL user your i...

Integrating Slack with Matillion ETL

Overview Slack is a popular messaging tool for teams to communicate and to automate notifications from third-party applications. Matillion ETL can send messages to Slack via Amazon SNS (Simple Notification Service) messages. This can be used...

Making Multiple API Queries

Overview Matillion ETL's API Query component is ideal for pulling data from a REST or SOAP API. However, there are cases where using a Python script to make the API call may be more efficient. As an example, suppose you have a list of addres...

Matillion Data Quality Framework

Overview The term “data quality” is very broad, and typically has a dual meaning: Data that is complete, accurate and relevant A set of processes to achieve such a state We’ll focus on the first of these two angles here, a...

NRT Replication In Redshift

Overview Amazon Redshift offers a fast and scalable data processing platform at around 1/10th the cost of a typical equivalent on-premises solution. This makes it perfect for integrating multiple different source systems, to produce timely, ...

Pivoting and Unpivoting Tables

Pivoting Data System databases can often hold data in a highly normalised form such as the below: However this format is not so user friendly and is not ideal for an end user BI tool. The Transpose Rows Component can be used with the Spl...

SCM Integration

Overview For many customers, best practice topology requires multiple instances of Matillion ETL (for development, testing and live) combined with a Source Control Management (SCM) system combined and build server or script environment to manage...

Tracking Loaded Files

Tracking Loaded Files A common use case for Matillion ETL is to use Matillion’s SQS integration to trigger the loading of a file from S3 into Redshift using lambda. Further details of this are here . Suppose some transformations should be ...

Turning a Backup of Matillion ETL into a new pre-release Instance

These instructions explain how to take a backup of an existing Matillion instance and then switch the new instance to the pre-release repository. This will then allow you to upgrade the instance to the pre-release version. Pre-release version...

Using Grid Variables to Apply Business Rules in a Transformation Job

Overview When Transforming data, it is often the case that business rules need to be applied to data to do different calculations on different rows of data depending on certain criteria. Traditionally these business rules would be hard coded...

Using incron to automatically copy data to S3

Overview This is not strictly related to Matillion ETL however it is a common task to get data easily into S3 from a premise or system.  This approach is especially used in a micro-batching scenario . The below instructions should be enou...

Using KMS encrypted passwords in Python

Overview Typical uses of Python scripts in Matillion ETL often involve the inclusion of credentials. It is always ill-advised to store a password in plaintext in a component. Unfortunately, Matillion ETL does not allow Python scripts to ta...

Using R with Matillion ETL for Redshift

Overview Matillion ETL for Redshift runs as an EC2 Linux instance in your AWS VPC. You have full administrative access to the EC2 instance, and can install additional software - such as the statistical package R . From within Matillion yo...

Using Table Metadata to Grid

​ Overview The Table Metadata To Grid component in Matillion ETL takes metadata from a table and uses this metadata to populate a grid variable . This page offers an example of how and why the component might be used. In this example we...

Managing Python on a Matillion ETL Virtual Machine (VM)

Overview Note This guide was provided by Matillion's solution architect team. Python is a popular programming language for data engineers. Recognizing this, Matillion ETL offers a Python Script component, which allows Matillion users t...

Cookie consent

We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content based on your browsing and navigation history.