Matillion ETL features a scheduler that will launch orchestration jobs automatically at a pre-defined regular time interval. Schedules are set up against a project and you can have multiple schedules set up. The scheduler is straight forward to set and based on Quartz technology.
Creating a Schedule
The scheduler is available from the Project → Manage Schedules menu.
Scheduled jobs can be added using the + button and removed using the X beside each schedule. They can also be run on demand using the 'Run Now' button. Selecting a schedule will reveal a brief report of this schedule's run history which can be expanded in a new window with the 'Full History' button as shown below.
Clicking the + button to create a new schedule will bring up a new set of options.
After giving your job a name, the time of execution can be entered by giving Hours and Minutes. For example, 4:15pm would be 16 hours and 15 minutes. A job can be run multiple times by entering multiple hours or minutes separated by commas. The wildcard character, *, can be used to signify every hour or every minute in their respective fields.
A time increment can be specified using / to separate two values. The first value denotes the number to begin from and the second is the number to increment. Thus 10/5 would mean to start at 10 with increments of 5 (10,15,20,25....). Thus, we can use these in the minutes and seconds fields to run a schedule at regular intervals. For example: 0/15 in the minutes field would mean to schedule the task every 15 minutes.
The job itself is run by Matillion ETL, which is always based on UTC and is offset to local time using the Timezone dropdown menu. The 'Next Run' for all job times shows a similar time offset but in the timezone of the browser, rather than the one selected in the dropdown menu. Recent changes to timezone and/or daylight saving rules may not be reflected until you perform an update of the instance that Matillion ETL is running on. Update instructions can be found here.
Example: In the above image, the client wants to schedule data backups at 3:15am CEST. Since Matillion ETL factors in daylight savings hours automatically, CET is chosen from the dropdown timezone list to represent CEST. In the 'Test' window, Matillion ETL shows this time and also the UTC time that it has been calculated from. However, the browser window currently being used is in British Summer Time (GMT+1) and so we see a time local to this browser in the 'Next Run' field at the top of the window. All 3 times (GMT+1, CEST and UTC) are the same, but relative to different zones.
Jobs can be scheduled to run on a weekly basis or indeed each day if all days are selected in Days Of Week. Similarly, numbers of days (separated by commas) can be entered into Days Of Month to run on a monthly basis. Use * to mean every day. Use L to indicate the last day of the month. You can also use an expression such as 15W to indicate the closest weekday to the 15th.
Note: Scheduled tasks cannot occur during the maintenance window of the EC2 cluster that Matillion ETL is running on. Inputting a time during the maintenance window will yield an error message when the schedule is tested.
The rest of the information can be selected from dropdown menus that read from the current Matillion ETL instance.
- Version: Select the version of the project you wish to run at the scheduled time.
- Job: Select the Orchestration job you wish to run at the scheduled time.
- Environment: Select the environment you wish to run against at the scheduled time.
You may test the schedule by pressing the Test Schedule button. If successful, it will indicate the next time the schedule will run. The test will not actually schedule anything but instead just checks the information in the form. Jobs are not scheduled until you confirm by pressing OK.
When a schedule runs, it runs a particular Orchestration Job found within a particular Version with a particular Environment, therefore please select each of these items before saving with the OK button.
You will see the tasks run in the Tasks panel
The scheduling system is based on Quartz, a powerful scheduling system with some advanced features that are covered in more detail here. Some special characters include:
Hint: If you choose a time-zone with daylight savings you make get unexpected results for schedules up to 3 hours after midnight on the day the clocks are adjusted.
Note that jobs can also be created for any given job by Right-Clicking the job in the Explorer panel and selecting Schedule Job.