Integrating Slack with Matillion ETL

Integrating Slack with Matillion ETL


Overview

Slack is a popular messaging tool for teams to communicate and to automate notifications from third party applications. Matillion can send messages to Slack via Amazon SNS messages. This can be used to create a Slack Channel to provide updates on the status of Matillion job runs.

 

Setting Up Slack

In order to configure Matillion to work with Slack, the first step is to setup a Slack channel to hold all of the Matillion messages and to configure it to listen to the SNS messages using an Incoming WebHook. To do this, search the Apps in Slack, which is accessed via the Slack Menu -> Apps & integrations. Search for Incoming WebHooks:

Add a new Configuration using the Add Configuration button.

Select the Slack Channel to post to or Create a new Channel:

In this example a channel has been created called “aws-sns”

Select Add Incoming WebHooks Integration to access the Setup Instructions. Make a note of the Webhook URL from below:

The next step is to Create an SNS Topic for Matillion to send the messages to. This is done in the AWS console. This example has a Topic called Slack created:



Setting Up AWS

1. To create a new Lambda function so SNS messages are pushed to Slack, click "Author from scratch", and enter a Function name that doesn't exist.


This will open the "Function code" section at the bottom of the screen.

Replace the sample code with the following:

console.log('Loading function');

const https = require('https');
const url = require('url');
const slack_url = '<webhook url>';
const slack_req_opts = url.parse(slack_url);
slack_req_opts.method = 'POST';
slack_req_opts.headers = {'Content-Type': 'application/json'};

exports.handler = function(event, context) {
(event.Records || []).forEach(function (rec) {
if (rec.Sns) {
var req = https.request(slack_req_opts, function (res) {
if (res.statusCode === 200) {
context.succeed('sns posted to slack');
} else {
context.fail('status code: ' + res.statusCode);
}
});

req.on('error', function(e) {
console.log('problem with request: ' + e.message);
context.fail(e.message);
});

req.write(JSON.stringify({text: `${rec.Sns.Message}`});

req.end();
}
});
};

Replace the "const slack_url" variable URL with the Webhook URL generated above.

Click "Save".

4. Now we need to Publish the Function.

From the drop-down menu at the top, select "Publish new version".



Click "Create".

6. Click the "Test" button, with the new Event name next to it in the drop-down box. You should now receive a message in your Slack channel.

7. The section below will show you how to set up Matillion. 
 

Setting Up Matillion

Matillion can now put messages on the SNS Topic and these messages will appear in Slack. An example job is shown below.

This Orchestration job runs a Transformation job. The Transformation job has been configured to export the Duration of the job to a Global Environment Variable called time.

Once the Transformation job runs successfully, Matillion puts a message on the SNS Topic and the message is that the job has successfully run and how long the job ran for:

This message is successfully posted to the Slack Channel