IAM Roles & Permissions (AWS)
  • Dark

IAM Roles & Permissions (AWS)

  • Dark


You manage access in AWS by creating policies and attaching them to IAM identities (users, groups of users, or roles) or AWS resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

AWS credentials are required for Matillion ETL instance to access various services such as discovering S3 buckets and using KMS. Appropriate permissions must be given via your AWS admin console and details of your GCP account must be entered into the Matillion ETL instance via Project Manage Credentials where credentials for other platforms may also be entered.

Please Note

For more information on Manage Credentials, please visit to the document here.

You can grant access to other AWS services using the Attached Policy File (AWSPolicy.txt)

Adding an AWS Access Policy

There are two ways you can add policy and give access to Matillion ETL.

  1. With instance credentials:This is done by specifying an IAM Role for the EC2 instance at launch time. Once in place, you can add and remove permissions from the chosen IAM role and the changes take effect immediately. This gives you fine grained control over exactly which AWS services Matillion ETL is allowed to use. This method is the easiest to manage if you attach the role to the instance while it is being launched. However it is possible to add an IAM to a running instance, using the AWS command line.
  2. With existing user defined credentials:These can be defined via the Matillion ETL user interface at any time. The credentials can be created via an IAM User, and Matillion will be able to access what the IAM user has access to. You can simply enter an Access Key ID and a Secret Access Key.
  3. Depending on your company’s security policy, you may choose to use either:

    • Coarse-grained access control: easier to administer, but typically grants many more privileges than are necessary.
    • Fine-grained access control: requires you to specify a more complex policy but follows the principle of least privilege. A template for this policy type is attached to the document here for the convenience.

AWS Coarse-grained access control

With this approach, you simply attach the following Managed Policies to the roles you are using.

Managed Policy Purpose
AmazonRedshiftReadOnlyAccess This policy allows Matillion to read metadata about the Redshift clusters, such as endpoint and port number. This makes it easier to set up new projects and environments. If you don't allow this role, you will need to type in all the information manually.
Note: It does not allow Matillion to read data from Redshift (that privilege is granted by you providing a username and password, and by you permitting network connectivity between Matillion and Redshift).
AmazonS3FullAccess This policy is used to read data to and from S3 buckets. It can be used in almost every load component, since they almost all either read directly from S3 or else use an S3 Staging area.
AmazonSNSFullAccess This policy is used when creating SNS topics and publishing messages to them. This is the purpose of the SNS Message Component .
AmazonSQSFullAccess This policy enables Matillion to read from an SQS queue, or to write an SQS message using the SQS Message Component.
CloudWatchFullAccess This policy is required to publish metrics to CloudWatch Publish component.
AmazonRDSReadOnlyAccess This policy is used by the RDS Bulk Output components to discover information about the available RDS servers in the region. Without this policy, Matillion may still be able to connect to RDS provided the network path is set up and a username and password have been provided.
AmazonEC2FullAccess This policy is used when performing automated backups.
SecretsManagerReadWrite Provides access to Secrets Manager operations. The policy doesn't allow the identity to configure rotation because rotation requires IAM permissions to create roles. To learn more, read AWS managed policies available for use with AWS Secrets Manager.

Please Note

There are no single Managed Policy which can be used to authorize KMS access in the Manage Passwords dialog through ProjectManage Password. Please use the KMS table in the fine-grained access control section below.

AWS Fine-grained access control

This section lists all the IAM privileges which Matillion ETL can require during normal operation. Privileges are shown using AWS policy action metamodel version "2012-10-17". It does not take into account any privileges required by custom AWS requests made in Bash or Python scripts.

There are no mandatory IAM Actions which Matillion absolutely requires. However the recommended ones listed in the next section are the minimum required for productivity and ease of use.

Recommended Actions Description
redshift:DescribeClusters During first login, allows Matillion to find Redshift clusters. Also used when creating a new environment. Note: this privilege is not required in order to use a particular Redshift cluster: it’s just useful to make setup faster and less error prone.
s3:ListAllMyBuckets Enables every S3 Staging Area drop-down list to work. Used in every Orchestration data load component.
Note: Matillion ETL can run without this if you wish to keep your buckets private. However, exploration and discovery of buckets from within the client will no longer work.
s3:ListBucket This permission allows the user to access files in the folder (and any sub-folders). Can be used in every Orchestration data load component.
s3:GetObject Enables Matillion to read S3 files. Used in every Orchestration data load component.
s3:PutObject Used by every Orchestration data load component to create temporary files in a nominated staging bucket.
The files are used temporarily in order to take advantage of Redshift’s bulk loader and transfer the data as quickly as possible into Redshift. This Action could be restricted to specific staging buckets.
s3:DeleteObject Used for the same purpose as s3:PutObject, to remove temporary S3 files created during data loads.
s3:GetBucketLocation Uses the subresource location to return a bucket's AWS region.

Automated backups can be enabled from the AdminManage Backups menu to the top right of the page. Backups require the EC2 actions detailed below.

EC2 Actions Description


Used during creation of the EC2 snapshot.


Used to tag the generated snapshot with a name.


Required to retrieve instance metadata


Required to retrieve instance metadata

This section lists the actions required in order to use the KMS option in the Manage Passwords dialog.
Note that in addition to the below, the chosen KMS key must be:
  • In the same Region as Matillion
  • Enabled

In the following tables, we outline actions for KMS, RDS, SQS, SNS and Cloudwatch required by Matillion to perform certain functions.

KMS Action Description
kms:ListAliases Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt Enables Matillion to store an encrypted password.
kms:Decrypt Enables Matillion to retrieve and use an encrypted password.
RDS Actions Description


Enables Matillion to list the endpoints of the chosen RDS database type. Note this privilege is not required in order to query data from any RDS instance.

SQS Actions Description.


Enables the operator to select an SQS queue in the main SQS Configuration window.


Required in order to monitor queues.


Enables Matillion to receive messages from SQS (e.g. using the SQS Message orchestration component).


Enables Matillion to receive messages from SQS and remove them from the queue.


Enables Matillion to send messages to SQS.

SNS ActionsDescription


Required in order to publish messages with SNS (e.g. Using the SNS Message  component).


Enables the operator to select from a list of existing SNS topics (such as when using the SNS Message  component.


If the user manually types in a topic name which doesn’t already exist in that region, Matillion tries to create the new topic, which requires this privilege.

CloudWatch Actions Description


Required to enable the CloudWatch Publish component to list existing metrics.


Required in order to publish metrics to Cloudwatch using Matillion’s CloudWatch Publish component.This Action also includes the authorization to create new Custom Namespaces and new Metrics.

Credentials for CDC

Below is a full list of the permissions a Matillion ETL instance requires to use CDC as described in the above sections. We provide a brief outline of what each permission allows Matillion ETL to do and which section of the CDC setup it pertains to.

Lambda Action Description Client Section
lambda:CreateFunction Create Lambda Functions Manage
lambda:DeleteFunction Delete Lambda Functions Manage
lambda:GetFunction Get information about the Lambda Function Manage
lambda:TagResource Add tags to a Lambda Function Manage
lambda:UpdateFunctionCode Update a Lambda Function's code Manage
lambda:UpdateFunctionConfiguration Modify the version-specifc settings of a Lambda function Manage
lambda:AddPermission Get information about the Lambda Function Manage
lambda:RemovePermission Revokes Lambda Function permission from an AWS Service Manage
lambda:GetPolicy Get the resource-based IAM policy for a Lambda Function CDC Monitor
SQS Action Description  
sqs:DeleteMessage Delete a message from a specified queue Manage
sqs:ReceiveMessage Retrieve up to 10 messages from a specified queue Manage
sqs:DeleteQueue Delete a queue specified by the QueueUrl, regardless of the queue's contents Manage
sqs:ListQueues (Optional) Allows discovery of available queues Manage
sqs:ChangeMessageVisibility Change the visibility timeout of a message in a queue Manage
sqs:GetQueueUrl Get the URL of an Amazon SQS queue Manage
DMS Action Description  
dms:CreateEndpoint Create a DMS endpoint CDC Task Config
dms:DeleteEndpoint Delete a DMS endpoint CDC Task Config
dms:StartReplicationTask Start a replication task CDC Monitor
dms:CreateReplicationTask Create a replication task CDC Task Config
dms:ModifyEndpoint Modify a DMS endpoint CDC Monitor
dms:DescribeConnections Get the status of the connections between a replication instance and an endpoint CDC Task Config
dms:DescribeReplicationInstances Get information on replication instances on your account (region-specific) CDC Task Config
dms:DescribeReplicationTasks Get information on replication tasks on your account (region-specific) CDC Task Config
dms:StopReplicationTask Stop a replication task CDC Monitor
dms:DeleteReplicationTask Delete a replication task CDC Monitor
dms:DescribeEndpoints Get information on DMS endpoints on your account (region specific) CDC Task Config
dms:TestConnection Tests the connection between a replication instance and a DMS endpoint CDC Task Config
S3 Permissions Description  
s3:GetBucketNotification Get the notification configuration of a bucket CDC Task Config
s3:ListBucket View folders inside S3 buckets  
s3:PutBucketNotification Set notification configuration of an S3 bucket CDC Task Config
s3:ListAllMyBuckets (Optional) Get a list of all owned S3 buckets CDC Task Config
IAM Permissions Description  
iam:ListRoles (Optional) Get a list of IAM Roles Manage
iam:PassRole Allows passing a role to an AWS service Manage
EC2 Permissions Description  
ec2:DescribeRegions (Optional) Get the regions currently available to the instance Manage


The DynamoDB data staging components require some credentials to function properly.

DynamoDB Actions Description


Return information about the table, including the current status of the table, when it was created, the primary key schema, and any indexes on the table.


Return an array of table names associated with the current account and endpoint.


Scan a table or index.