Create Project
Contents
x

Create Project

Overview

In Matillion ETL, a "project" is a logical grouping of configuration settings and resources (such as Orchestration and Transformation Jobs). To use a Matillion ETL instance, a project must be created upon first logging in, or when switching to a new project. This guide describes how to create a project in Matillion ETL using the Create Project wizard – going into detail about each step.

Important Information

  • Before using this guide, a Matillion ETL instance must have been launched successfully.
  • The Create Project wizard will appear upon launching an empty Matillion ETL instance, or it can be accessed via the Project dropdown menu at the top left of the screen by clicking Project → Switch Project → Create Project.
  • There are no practical limits to the number of projects users can create. However, only one project is used by the Matillion ETL client session at a time, and each project must have a unique name.

Creating a project in Matillion ETL for Redshift

  1. In the Project Details step, the local Matillion ETL details of the project will be setup. Provide details for the following fields:

    • Project Group – provide a name for a new project group or select an existing project group from the dropdown list (this will simply be used to group similar projects logically)
    • Project Name – provide a name for the new project
    • Project Description – provide a searchable and clear description for the project (this field is optional)
    • Include Samples – tick this checkbox to include sample jobs in this project (this is ticked by default), then click Next

    Project Details

    Project Details

  2. The project requires an environment to make use of external resources. In the AWS Connection step, the details used to connect the project to the AWS cloud platform will be setup. Provide details for the following fields:

    • Environment Name – provide a name to describe the new Matillion ETL environment (click here to read more about environments)
    • AWS Credentials – select instance credentials, a set of existing AWS credentials from the dropdown list, or click Manage to add a new set of AWS credentials to use with the project (click here to read more about managing credentials), then click Next

    Please Note

    Choosing to use Instance Credentials rather than configuring specific credentials may limit the ability to correctly connect to resources.

    AWS Connection

    AWS Connection

  3. In the Redshift Connection step, the details used to connect the project to a Redshift cluster will be setup. Provide details for the following fields:

    • Redshift Cluster – if an IAM Role has been created, this box will list the Redshift clusters that are visible to that role within the region the Instance is running (selecting a cluster will populate a number of the below fields with sensible defaults)
    • Endpoint – provide the URL of the Leader Node
    • Port – provide the port number for the above endpoint (Redshift typically communicates over port 5439 or 5432)
    • Username – provide the username for the environment connection
    • Password – provide the password for the environment connection
    • Enable SSL – tick this checkbox to encrypt communications between Matillion and Redshift (this is required by some clusters), then click Next

    Redshift Connection

    Redshift Connection

  4. In the Redshift Defaults step, a connection to a database on the Redshift cluster will be setup. Provide details for the following fields:

    • Database – provide the name of the database on the specified Redshift cluster to use as default
    • Default Schema – provide the name of the schema to use as default (public is set as default)
    • Default Bucket – select an s3 bucket where project data will be staged as default
    • Concurrent Connections – select the number of concurrent connections that will be allowed (this feature is only available in Enterprise Mode, click here to read more about multiple environment connections), then click Finish

    Please Note

    It is recommended that all details are first tested before creating the project. To do this, click Test.

    Redshift Defaults

    Redshift Defaults

Creating a project in Matillion ETL for Snowflake

  1. In the Project Details step, the local Matillion ETL details of the project will be setup. Provide details for the following fields:

    • Project Group – provide a name for a new project group or select an existing project group from the dropdown list (this will simply be used to group similar projects logically)
    • Project Name – provide a name for the new project
    • Project Description – provide a searchable and clear description for the project (this field is optional)
    • Include Samples – tick this checkbox to include sample jobs in this project (this is ticked by default), then click Next

    Please Note

    While the names of steps and the fields within may differ between Cloud Data Warehouses, the process remains the same.

    Project Details

    Project details

  2. The project requires an environment to make use of external resources. In the AWS / Cloud / GCP Connection step, the details used to connect the project to the chosen cloud platform will be setup. Provide details for the following fields:

    • Environment Name – provide a name to describe the new Matillion ETL environment (click here to read more about environments)
    • AWS / Azure / GCP Credentials – select instance credentials, a set of existing cloud platform credentials from the dropdown list, or click Manage to add a new set of cloud platform credentials to use with the project (click here to read more about managing credentials), then click Next

    Please Note

    Choosing to use Instance Credentials rather than configuring specific credentials may limit the ability to correctly connect to resources.

    AWS / Cloud / GCP Connection

    AWS / Cloud / GCP Connection

  3. In the Snowflake Connection step, the details used to connect the project to the Snowflake data warehouse will be setup. Provide details for the following fields:

    • Account – provide the name of a Snowflake account (this is found at the beginning of the account's URL and may also include a region ID and cloud platform – for example, https://AccountName.eu-central-1.azure.snowflakecomputing.com)
    • Username – provide a username for the Snowflake account that has the required roles and privileges
    • Password Type – select to use either a Password or Private Key from the dropdown list (password is set as default)
    • Password – provide the password for the above Snowflake account
    • Connection Settings – click Manage to add connection settings (defined by parameter=value pairs), then click Next

    Please Note

    Snowflake for AWS also supports the use of key pair authentication. If Private Key is selected as the Password Type, the Password field will be replaced by the following fields:

    • Private Key – provide the private key for the Snowflake account
    • Passphrase – provide the passphrase linked to the above private key
    Snowflake Connection

    Snowflake Connection

  4. In the Snowflake Defaults step, a connection to a database in the Snowflake data warehouse will be setup. Provide details for the following fields:

    • Default Role – select the Snowflake Role the project with use as default (this field can be left empty if no changes are required)
    • Default Warehouse – select the cloud data warehouse on the Snowflake account to use as default from the dropdown list
    • Default Database – select the database on the Snowflake account to use as default from the dropdown list)
    • Default Schema – select the name of the schema to use as default from the dropdown list
    • Concurrent Connections – select the number of concurrent connections that will be allowed (this feature is only available in Enterprise Mode, click here to read more about multiple environment connections), then click Finish

    Please Note

    • New roles, warehouses, databases and schemas must be created from within the Snowflake account.
    • It is recommended that all details are first tested before creating the project. To do this, click Test.
    Snowflake Defaults

    Snowflake Defaults

Creating a project in Matillion ETL for BigQuery

  1. In the Project Details step, the local Matillion ETL details of the project will be setup. Provide details for the following fields:

    • Project Group – provide a name for a new project group or select an existing project group from the dropdown list (this will simply be used to group similar projects logically)
    • Project Name – provide a name for the new project
    • Project Description – provide a searchable and clear description for the project (this field is optional)
    • Include Samples – tick this checkbox to include sample jobs in this project (this is ticked by default), then click Next

    Project Details

    Project Details

  2. The project requires an environment to make use of external resources. In the Environment step, the details used to connect the project to the Google Cloud Platform will be setup. Provide details for the following fields:

    • Environment Name – provide a name to describe the new Matillion ETL environment (click here to read more about environments)
    • GCP Credentials – select instance credentials, a set of existing GCP credentials from the dropdown list, or click Manage to add a new set of GCP credentials to use with the project (click here to read more about managing credentials)
    • Default Project – provide the name of a project on the GCP account to use as default
    • Default Dataset – provide the name of a dataset on the GCP account to use as default, then click Finish

    Please Note

    • Default Datasets can be found via the Google Cloud Console → BigQuery → Resources.
    • It is recommended that all details are first tested before creating the project. To do this, click Test.

    Error

    Choosing to use Instance Credentials rather than configuring specific credentials may limit the ability to correctly connect to resources.

    Environments

    Environment

Creating a project in Matillion ETL for Azure Synapse

  1. In the Project Details step, the local Matillion ETL details of the project will be setup. Provide details for the following fields:

    • Project Group – provide a name for a new project group or select an existing project group from the dropdown list (this will simply be used to group similar projects logically)
    • Project Name – provide a name for the new project
    • Project Description – provide a searchable and clear description for the project (this field is optional)
    • Private Project – tick this checkbox to allow only users given access to view and work with the project.
    • Include Samples – tick this checkbox to include sample jobs in this project (this ticked by default), then click Next

    Project Details

    Project Details

  2. In the Cloud Connection step, the details used to connect the project to the Azure cloud platform will be setup. Provide details for the following fields:

    • Environment Name – provide a name to describe the new Matillion ETL environment (click here to read more about environments)
    • Azure Credentials – select instance credentials, a set of existing Azure credentials from the dropdown list, or click Manage to add a new set of Azure credentials to use with the project (click here to read more about managing credentials), then click Next

    Please Note

    Choosing to use Instance Credentials rather than configuring specific credentials may limit the ability to correctly connect to resources – for example, the Azure Blob Storage Load component relies on credentials with access to Blob Storage.

    Cloud Connection

    Cloud Connection

  3. In the Azure Synapse Connection step, the details used to connect the project to the Azure Synapse cloud data warehouse will be setup. Provide details for the following fields:

    • Endpoint – provide the URL of an existing Azure Synapse account endpoint
    • Port – provide the port number for the above endpoint (Azure Synapse typically communicates over port 1433)
    • Username – provide the username linked to the above Azure Synapse account
    • Password – provide the password linked to the above Azure Synapse account username
    • Connection Settings – click Manage to add connection settings (defined by parameter=value pairs), then click Next

    Please Note

    Azure Synapse credentials can be found via the Azure Portal → SQL Servers → Properties (Refer to this article for more details).

    Azure Synapse Connection

    Azure Synapse Connection

  4. In the Azure Synapse Defaults step, the default selection in the project will be setup. Provide details for the following fields:

    • Default Database – provide the name of the database on the specified Azure Synapse account to use as default
    • Default Schema – provide the name of the schema to use as default
    • Concurrent Connections – select the number of concurrent connections that will be allowed (this feature is only available in Enterprise Mode, click here to read more about multiple environment connections), then click Finish

    Please Note

    It is recommended that all details are first tested before creating the project. To do this, click Test.

    Azure Synapse Defaults

    Azure Synapse Defaults


Next Steps

If this is the first time this Matillion ETL instance has been launched, it is strongly advised the first action undertaken on the client is to replace the default username and password. This can be done via the User Configuration window found under the Admin dropdown menu.