Integrating Slack with Matillion ETL
  • Dark
    Light
  • PDF

Integrating Slack with Matillion ETL

  • Dark
    Light
  • PDF

Overview

Slack is a popular messaging tool for teams to communicate and to automate notifications from third-party applications. Matillion ETL can send messages to Slack via Amazon SNS (Simple Notification Service) messages. This can be used to create a Slack channel that provides updates on the status of Matillion ETL job runs.



Setting Up Slack

To configure Matillion ETL to work with Slack, the first step is to set up a Slack channel to hold all of the Matillion ETL messages, and to configure it to listen to the SNS messages using an Incoming WebHook Slack app.

1. From the Slack menu, access the Apps page and search for Incoming WebHooks:

Incoming WebHooks

2. Click Add to Slack to add a new Incoming WebHooks configuration.

3. Select an existing Slack channel to post to, or click create a new channel if the channel you want to use doesn't already exist.

Select Slack channel

In this example a channel called "aws-sns" has already been created in Slack.

4. Click Add Incoming WebHooks Integration to access the setup instructions. Copy or make a note of the Webhook URL as you will need this later. You do not need to configure anything on this page, so leave the defaults as they are and click Save Settings.

The next step is to create an SNS topic for Matillion ETL to send the messages to. This is done in the AWS console, as explained in the next section.




Setting Up AWS

In AWS you will create an SNS topic for Matillion ETL to send messages to, and a Lambda function to control pushing the SNS messages to Slack.

1. In the AWS console, select Simple Notification Service and then select Create Topic.

2. Select Standard as the topic type, and give the topic a unique name (e.g. "Slack"). You do not need to configure anything else on this screen, so click Create topic to proceed.

Create topic

3. Copy or make a note of the topic's ARN, as you will need this when configuring the Lambda function.

4. In the AWS console, select Lambda, → Create functionAuthor from scratch.

5. Enter a name for the function. This must be a unique name that you have not used for any other Lambda function.

6. You need an execution role with basic permissions. By default a new role with basic Lambda permissions will be created, so you can simply confirm that this option is selected and then click Create function. Alternatively, you can use an existing role that has been created with appropriate permissions, or create a new role from AWS policy templates. You can learn more about these options in the AWS documentation.

Select role creation options

7. The next step is to add an SNS trigger. Click Add Trigger, and then choose SNS from the dropdown list of trigger options. Paste the ARN of the SNS topic you created previously, and then click Add.

Trigger configuration

8. On the Code tab, replace the sample code in the Code source box with the following:

console.log('Loading function');

const https = require('https');
const url = require('url');
const slack_url = '<webhook url>';
const slack_req_opts = url.parse(slack_url);
slack_req_opts.method = 'POST';
slack_req_opts.headers = {'Content-Type': 'application/json'};

exports.handler = function(event, context) {
    (event.Records || []).forEach(function (rec) {
        if (rec.Sns) {
            var req = https.request(slack_req_opts, function (res) {
                if (res.statusCode === 200) {
                    context.succeed('sns posted to slack');
                } else {
                    context.fail('status code: ' + res.statusCode);
                }
            });
            req.on('error', function(e) {
                console.log('problem with request: ' + e.message);
                context.fail(e.message);
            });
            req.write(JSON.stringify({text: `${rec.Sns.Message}`}));
            req.end();
        }
    });
};

Replace the "const slack_url" string in this code with the Webhook URL generated when you configured the Slack WebHook app.

Enter code source

9. Click FileSave from the menu at the top of the Code source box.

10. Now you need to publish the function. Click Actions in the top-right of the screen and then click Publish new version. Optionally, enter a description of the version being published (for example, "Matillion to Slack initial version"), and then click Publish.

11. Copy the Function ARN as you will now need to create a subscription on your SNS topic that uses this.

12. Return to the Amazon SNS dashboard and select the topic you previously created. Click Create subscription.

13. For Protocol, select AWS Lambda, and paste the ARN of your Lambda function into the Endpoint field.

Configure subscription

14. Click Create subscription.

You can now test the function to verify that it will send messages to your Slack channel. You do this as follows.

1. Select the Test tab.

2. Select SNS Topic Notification in the Template dropdown field and enter a suitable name in the Name field.

3. In the code field, replace both the "EventSubscriptionArn" and "TopicArn" parameters with the ARN of the SNS topic you have previously created, and enter a suitable test message as highlighted here:

Configure test message

4. Click Create.

5. Either on the Test tab or back on the Code tab, click the Test button and then verify that the test message has been sent to your chosen Slack channel.

You can now go on to configure Matillion ETL to use this SNS topic to send messages to Slack.



Setting Up Matillion ETL

Matillion ETL can now send messages to the SNS topic, and these messages will appear in Slack. An example job that will output messages is shown below.

Example Matillion ETL job

This Orchestration Job runs a Transformation Job. The Transformation Job has been configured to export the duration of the job to a global environment variable called "time".

Job export variable

The Post To Slack component is an SNS Message component that has been configured to put a message on the SNS "Slack" topic created previously, as follows.

Post to Slack component configuration

  • Topic Name: The name of the SNS topic configured previously, in this example "Slack".
  • Message: The message that you want to post to your Slack channel when this job runs, for example:

Example SNS message content

This example would post the following message to the the Slack channel:

Example of posted message