Pipelines are configurable objects inside Matillion Data Loader that are responsible for loading data from a single source service to a single target data warehouse. For information on creating a new Pipeline, see Adding a Pipeline.
The Manage button beside each Pipeline enables you access a detailed view for the pipeline and make changes to the configuration. The pipeline is opened in a new tab, allowing you to manage several pipelines at once and return to the homescreen without closing the window.
Please do not alter the tables created by Matillion in your target cloud data warehouse (e.g. renaming or changing metadata types). Doing so could prevent your Pipelines from running properly.
A number of options are available at the top-left of the window of each pipeline tab:
- Enable/Disable: Select the toggle to Enable or Disable your pipeline. A newly created pipeline is enabled by default.
- Sync Now: Allows you to run the pipeline immediately, independently of the current schedule.
- Refresh: Update the pipeline to include any recent changes.
- Info: Technical information for the pipeline.
- Schedule: View and change the pipeline's schedule. Please note that this change takes effect from the time you make it, not from the time of the last scheduled execution of the pipeline.
- Edit: View and modify certain pipeline and data source settings. Please note, most - but not all - pipelines are Schema Drift compatible. Please refer to each relevent data source's topic for more information.
- Delete: Remove the pipeline. This does not remove any shared properties, such as user credentials. This action cannot be reversed.
You can also click the Info button beneath each run to view a detailed log of that run's performance. There is also a detailed graphical and tabular history of the pipeline's performance below the information panel. Refer to the Metrics Dashboard article for more information.