UI And Basic Functions
This a quick introduction to the user interface (UI) and basic functionality of Matillion ETL. While there are a wide array of functions and options available, the goal of this guide to get new users up and running simple tasks in minutes. It will discuss the basics of Matillion ETL, explain common terminology and show how to use general interface features.
- Before beginning, a working instance of Matillion ETL will need to be running. Please refer to Launching a Matillion ETL Instance if help is required setting up an instance.
- While some features may differ between Cloud Data Warehouses, functionality of the Matillion ETL instance remains the same.
The majority of Matillion ETL features can be accessed using the following system features:
- Project Menu
- Navigation Panel
- Job Canvas
- Components, Shared Jobs and Environments Panel
- Properties Panel
- Tasks Panel
- Admin Menu
- Help Menu
1. Project Menu
The Project dropdown menu provides a list of options regarding the current Project. These options include:
- Switch Project: browse to or create a project (jobs inside other projects can also be opened from here, automatically switching to their parent project)
- Manage Project: set a name, description and permissions for sharing the current project (please refer to Project User Access for more details)
- Rename Project Group: change the name of the group associated with the current project
- Delete Project: delete the current project
- Input Data Report: generate an Input Data Report on all jobs run in the current session within the current project
- Manage Versions: create, delete and switch versions, or change the details of the current version (default and locked versions cannot be renamed, and current versions can only be renamed if more than one version is available—please refer to Manage Versions for more details)
- Recycle Bin: delete the current version (default and locked versions cannot be deleted)
- Add Environment: add an environment (Please refer to Environments for more details)
- Manage Environment Variables: define variables for the current environment (Please refer to Environment Variables for more details)
- Export: export selected jobs, environments and variables for use in other projects (exported as .json file—please refer to Import - Export for more details
- Import: import .json files containing jobs, environments and variables for use in the current project
- Manage Credentials: configure credentials to be used to integrate with the Matillion ETL instance environment (please refer to Manage Credentials for more details)
- Manage Schedules: schedule Matillion ETL jobs (please refer to Manage Schedules for more details)
- Task History: show a detailed list of successfully and unsuccessfully completed tasks (please refer to Task Management for more details)
- Manage OAuth: configure OAuth for various APIs from which Matillion ETL will source data (please refer to Manage OAuth for more details)
- Manage Passwords: configure username-password pairs to be stored for use in different parts of Matillion ETL (please refer to Manage Passwords for more details)
- Manage API Profiles: configure profiles to describe JSON- or XML-based API for importing data into tables (please refer to Matillion ETL API and Manage API Profiles for more details)
- Manage Webhook Payloads: manage webhook payloads in Matillion ETL for use with the Webhooks component (please refer to Manage Webhook Payloads for more details)
- Manage Shared Jobs: bundle workflows into a single custom component and use those components elsewhere in the project (please refer to Managed Shared Jobs for more details)
- Manage SQS Configuration: configure Matillion ETL with Amazon Simple Queue Service (SQS) (please refer to Manage SQS Configuration for more details)
- Manage Change Data Capture: synchronise Cloud Data Warehouse data with the local database to ensur all data is continually and automatically kept up to date (please refer to Manage CDC for more details)
- Manage Pub/Sub Configurations: enable and configure Pub/Sub services when using a Matillion ETL as part of a larger process (please refer to Manage Pub-Sub Configuration for more details)
- Migrate: send Matillion ETL resources from one Matillion Instance to another (please refer to Migrate for more details)
- Git: initialise a local Git repository or clone a non-Matillion repository to be integrated into the Matillion ETL instance (please refer to Git Integration for more details)
- Logout: log the current user out of the current Matillion ETL instance
Administration and Authentication
2. Navigation Panel
Each Matillion ETL project is made up of "jobs". All available jobs are displayed within this panel. There is no limit to the number of jobs that may be contained within a project. Double clicking a job on the list will open it on the Job Canvas. Alternatively, right clicking anywhere within the panel will open a pop-up menu which allows the user to create, delete, import and export jobs, as well as creating job folders.
There are two kinds of job:
- Orchestration Job: generally loads and unloads data to tables and file storage, also known as staging the data
- Transformation Job: generally manipulates data in the form of tables
Each job is made up of a series of tasks each represented by a "component". Multiple components can also be linked together to tackle more sophisticated tasks. Each type of job has its own set of components that can only be used within their respective job canvas. Orchestration Jobs may also be linked to Transformation Jobs, allowing large pipelines to be planned and executed using a single job.
3. Job Canvas
The job canvas shows the contents of a selected job. Each open job will be represented by a tab above the job canvas. Open jobs can be toggled between by clicking on these tab, without affecting other open jobs.
When creating a new job, the job canvas will be empty—ready for the user to fill with components. These components can be found in the Components panel and dragged onto the job canvas.
These components will then need to be linked to one another in the order in which they will need to be executed. This can be done by clicking on a component icon on the job canvas to reveal the connection rings (represented by ) around the component. Clicking and dragging on a ring will extend a connection that can be affixed to another component. These rings represent four potential connection options:
- Green / Success: Continue on to the next component if executed successfully
- Red / Failed: Continue on to the next component if failed to execute
- Grey / Unconditional: Continue on to the next component regardless of success or failure
- Blue / Iterate: Continue on to another component (regarding iterator components)
- Thses connection options allow users to direct the job according to whether the current run is successful or not.
- All Orchestration Jobs must begin with a Start component and thus, cannot receive links from other components.
When using multiple components, a job canvas can quickly become quite clutter or complex. To aid in understanding these jobs, users can add notes by right clicking anywhere on the job canvas, then clicking Create Note. Notes can be used to:
- Add explanations of functions, components and workflows directly on the job canvas
- Create blocks to group multiple components together to make sense of large workflows
- Provide information for other users working on the same job
4. Components, Shared Jobs and Environments Panel
This is a collapsible list of all components available to the currently-selected job type. Switching between an Orchestration Job and a Transformation Job will automatically update the available components, such that it is impossible to place a Transformation component into an Orchestration job or vice versa. A search field at the top of the panel allows the user to search for specific components (matching components by name only).
Each component embodies a particular function—from moving data into file storage to performing custom calculations on table columns. Clicking and dragging a component onto the job canvas will create a component of that type in the current job. There is no limit to the number of components that can apply to a job.
Components are named according to their function, although it is common for a single component to have multiple functions that are not strictly related to one another. Thus, it is recommended that users acquaint themselves with the available components such that they can make informed decisions on which components to use to best achieve a given goal.
Shared Jobs Panel
Shared jobs allow users to bundle entire workflows into a single custom component that can then be used anywhere else in the project. These custom components and other pre-made shared jobs are listed in the Shared Jobs panel, and may be dragged onto the job canvas and used in the same way as a normal component. Please refer to Shared Jobs for more details.
In Matillion ETL, an environment describes a single connection to a database, on which jobs can be run. Numerous environments can be set up at once, meaning different users can use different environments on a single Matillion ETL instance. All environments associated with the current instance are listed in this panel, along with the schemas, tables and columns contained within each environment. Please refer to Environments for more details.
5. Properties Panel
Clicking a component icon on the job canvas reveal detailed information about the component in the Properties panel below. Components generally share a similar set of information tabs, but with different contents in each. Common tabs include:
- Properties: a unique set of user-defined properties that effect the function of the selected component—hanges made to these properties will not affect the properties of any other component, even components of the same type (please refer to the specific component's documentation for more details)
- Export: export certain values from the selected component as a variable to be used elsewhere (please refer to Variables for more details)
- Sample: many components allow users to produce a sample output of data up to a specified number of rows—this can be used to confirm a component is set up correctly before using it in a live job
- Metadata: display data for each column featured in the sample data; including column names, types, sizes and scales (this tab is only available in Transformation components)
- Lineage: (Enterprise Mode Only) track columns across transformations in order to understand how that column has been affected by components up to the point of inspection
- SQL: display the SQL equivalent of the component and its current setup
- Plan: get the plan from the SQL query generated by the job up to (and including) the selected component
- Help: display a snippet of the help documentation for the selected component—including an explanation of its function and properties, as well as a link to the full documentation
6. Tasks Panel
The Task Tabs show information about tasks that have been run during the current session. Right clicking anywhere in the Canvas, the user has the option to run the current job as it currently stands. Right clicking on a component will also allow the user to run just that single component. In both cases, the run takes the form of a task that is documented in the Task Tabs. For more information on tasks, see documentation for Task Management. These tabs include:
- Tasks: a list of up to the last 20 tasks attempted by the current user in the current project session—including queued, failed or cancelled tasks (please refer to Task Management for more details)
- Search: allows users to search the current project for matching components (Please refer to Search Tab for more details)
- Console: the Matillion ETL instance console
- Command Log: an archive of interactions between the current browser session and the Matillion ETL server
- Notices: a list of notifications concerning the Matillion ETL project—including updates to Matillion ETL
7. Admin Menu
The Admin dropdown menu allows users with "Admin" privileges to manage the Matillion ETL instance (please refer to Admin Menu for more details). The following options are available:
- Download Server Log: download the available Tomcat Catalina server logs of the Matillion ETL instance
- Matillion ETL Updates: check for available updates for Matillion ETL (please refer to Updating Matillion ETL for more details)
- User Configuration: configure and manage user privileges and access rights (please refer to User Configuration for more details)
- SSL: configure the Matillion ETL instance to listen for HTTP and HTTPS requests
- Restart Server: restart the current Matillion ETL instance
- Manage Database Drivers: upload JDBC Drivers to use with the Database Query components (please refer to Manage Database Drivers for more details)
- Manage Backups: configure automatic backups and test backup schedules (please refer to Manage Backups for more details)
- Manage Groups: (Enterprise Mode Only) create, edit and remove user groups
- Manage Permissions: (Enterprise Mode Only) adjust the permission settings for user groups
- Audit: view a log of all activity on the Matillion ETL instance (please refer to Audit Log for more details)
- Product Improvement Metrics: choose to allow the collection of anonymous usage data to help Matillion build a better product (please refer to Product Improvement Metrics for more details and to see a list of what data is collected)
8. Help Menu
The Help dropdown menu provides a lists useful information about Matillion ETL, these include:
- Get Support: open a new browser window with contact details and information on Matillion ETL support
- Support Information: provides a template to send advanced enquiries to Matillion ETL support (please refer to Getting Support for more details)
- Manage Connections: manage multiple connection to the same Matillion ETL instance to enable users to see what other users are doing within the instance (please refer to Manage Connections for more details)
- License: view and manage Matillion ETL licenses
- About: display information about the current Matillion ETL version
If more than one user is currently logged into a project, these users will be represented by boxes alongside Admin and Help . Hovering over one of these boxes will display that user's current job status.
There are several common Windows keyboard shortcuts can be used inside Matillion ETL, these include:
- Ctrl + C: copy the selected component, job or text
- Ctrl + V: paste a copied component, job or text
- Ctrl + Z: undo the last change made to the job canvas (this can be done multiple times to revert back to any state within the current session, however this only applies to changes made on the job canvas)
- Ctrl + Y: redo the previous "undo" change made to the job canvas (this can only be used after Undo, but then be used multiple times until the latest version is reached)
- Ctrl + A: select all components, notes and links on the job canvas
- Ctrl + Return: if on the job canvas, the job will attempt to run, alternatively, if a pop-up window is open, the window will close
- Ctrl + Shift + C: open a search window listing all jobs in the current project
- Ctrl + T: open the Search tab in the Tasks panel
A reminder of the above keyboard shortcuts can be found on the job canvas menu, displayed when right-clicking on the job canvas.
Running A Job
Jobs are automatically saved with each change, so there is no need to manually save the progress—meaning they can be closed and reopened at any time to continue editing. At no stage will a job reach a "final" state where it cannot be further iterated upon.
Jobs can be run at any stage by simply right-clicking on the job and clicking Run on the job canvas menu. Alternatively, jobs can also be scheduled to run at a future time to queued behind other jobs. Please refer to Manage Schedules for more details.