Migrate
  • Dark
    Light

Migrate

  • Dark
    Light

Overview

Migrate allows you to send Matillion ETL resources from one Matillion ETL instance (the "source") to another (the "target"). When a resource is migrated, the resource is essentially "cloned" to the target instance. No resources on the source machine are destroyed as a result of migration.

Caution

Users should not migrate from a DBT-enabled instance to a non-DBT-enabled instance.


Preparing to migrate

When preparing to migrate from one instance to another, please consider the following:

  • While the risk of issues during a migration is much smaller than performing an in-place upgrade, it is still prudent to take a backup of your Matillion ETL instance before migrating.
  • If you do not have a target Matillion ETL instance (where you are migrating resources to) created already, you will need to launch it before migrating.
  • You must ensure that the target Matillion ETL instance:
    • Is running a version of Matillion ETL that is identical to or newer than the source instance. For example, if the source instance is running version 1.53, then the target instance must be running version 1.53 or newer.
    • Is on the same cloud data warehouse service as the source instance. For example, Snowflake to Snowflake.
    • Is hosted on the same marketplace platform as the source instance. For example, AWS to AWS.
    • Is given the same cloud privileges as the original, including feature access rights and firewall rules.
    • Allows incoming TCP/IP connections from the source Matillion ETL server.
    • Has the hostname packages.matillion.com whitelisted, to allow the target instance to update.
  • Please also ensure that:
    • Any connections are secured so that data is protected while in transit.
    • The user credentials (see User Configuration) that you plan to use to authenticate to each instance (source and target) have the necessary server-level user roles assigned, as follows:
      • API (this applies to all migrations)
      • Global Project Admin (applies when migrating project content such as Orchestration Jobs, Transformation Jobs, environments, passwords, queues, schedules, variables, versions, CDC config, and CDC tasks).
  • All users should be logged out when performing a migration. This is recommended to avoid any issues with changes to resources that are actively being worked on, or any credentials that are in use.

Note

Permissions still apply as usual to both the source user and target user. For example, if the target instance user does not have permission to write Shared Jobs, then that user cannot import Shared Jobs via the Migrate wizard. Read Manage Permissions for more information on how these are set.


Resources that are not automatically migrated

The following resources can't currently be migrated via Matillion ETL's Migrate feature:

  • Custom licences
  • SSL certificates
  • Task history
  • Python libraries
  • Non-standard JDBC drivers (click Admin in the top-right of the user interface (UI), and then click Manage Database Drivers to view the list of database drivers that come with Matillion ETL "out of the box").
  • Customized "standard" API profiles.
  • LDAP setup
  • Backend configuration changes (for example: Emerald.properties file changes)

Note

  • It is recommended that you configure any of the above non-migratable resources on your target instance before the migration.
  • If you rely on operating system (OS) customizations, such as additional libraries or drivers, please consider creating your own AMI based on ours with your customizations "baked in". Then, upon each release, remake your AMI from our latest one and test. This could also be used to ensure that drivers, user configuration, and so on, are "baked in".



Migrating Git resources

As of version 1.61, Matillion ETL supports the migration of Git resources. You can choose to migrate project content with or without Git, as described below.

When you perform a migrate with Git enabled, commits for orchestration and transformation jobs in every project in the target instance will be flagged as Modified. You will need to perform a one-time commit of all jobs after the migrate, which will remove the Modified flag.

This is expected behavior, due to some of the formats of the jobs changing during the upgrade, and is not a cause for concern. These are valid changes to the structure of the jobs and you can commit them safely.

To learn more about Git integration within Matillion ETL, read Git Integration with Matillion ETL.


Migrate: Connection

To use Migrate, click AdminMigrate in the Matillion ETL instance you are migrating from.

The first page of the Migrate wizard seeks to establish a connection to the target Matillion ETL instance (i.e. where you will migrate resources to). The properties are described in the table below.

Property Description
Target instance URL The URL of the Matillion ETL instance you wish to migrate resources to. You must include the "http://" or "https://" portion of the URL.
Username Your Matillion ETL username for the target instance.
Password Your Matillion ETL password for the target instance.

You must click Test Connection to confirm that a connection has been established before you are able to move on in the wizard. Once the test is completed successfully, click Next.


Migrate: Users

The second page of the wizard concerns the migration of user configuration, including users, user groups, usernames, passwords, roles, and permissions.

Users who wish to migrate user resources must have the role Server Admin for their source user and their target user.

A user's username and password are securely migrated from source to target.

LDAP must be set up in the target instance before the migration. Read LDAP Integration to learn more.

Caution

All user, group and permissions data will be overwritten in the Target instance if it already exists.

Select Migrate all users, groups and permissions if you want to go ahead and migrate your users.

Click Next to move to the next page.


Migrate: Project Content

The third page offers a nested display of your source instance's Project Groups. Explore the nested structure and select the project content that you wish to migrate.

If your Matillion ETL instance has projects that use Git, this page of wizard will ask How would you like to migrate projects which use Git? Select either Migrate with Git or Migrate without Git. The projects that are using Git are identified in the wizard by a Git icon, , next to the name.

Migrating projects with Git will migrate the local Git repository, including remote repository configuration if it exists. Any uncommitted changes will not be migrated.

Note

  • If migrating only part of a project group or project, please ensure that the relevant hierarchy exists on the target instance. For example, if migrating a version, the project and project group must already exist.
  • If you are migrating project content, the source and target users must have the Global Project Admin role in Matillion ETL. See User Configuration for details of how to do this.
  • Projects are filtered based on an access control list, and you will not be able to migrate a project unless you can access it on both the source and target instances.

Click Next to move to the next page.


Migrate: Instance Configuration

The fourth page offers a nested display of instance-level configuration resources. Explore the nested structure and select the resources you wish to migrate.

Click Next to move to the next page.


Migrate: Migration Options

The fifth and final page lets you disable schedules, queues, CDC tasks, and delete existing projects if you wish.

  • Disable schedules: Will disable the scheduler on the target instance. This is selected by default. This will not affect individual schedules, which will retain their Enabled/Disabled status, but schedules will not run until the scheduler is re-enabled. Users can re-enable the scheduler at their own discretion.
  • Disable queues: Will disable queues on the target instance. This is selected by default. Users can re-enable their queues at their own discretion.
  • Disable CDC: Will disable Change Data Capture (where available) on the target instance. This is selected by default. Users can re-enable CDC at their own discretion.
  • Disable CDC task source creation: Will disable CDC task source creation (where available) on the target instance. This is selected by default.
  • Delete project if already exists: Select this if you are migrating a project that already exists on the target instance in some form and you wish to delete the existing version of the project on the target instance. This is not selected by default.

When you are happy with the configuration, click Migrate to begin the migration process.

On completion, a Migrate dialog will show the status of the migration, listing each migrated element with the message "Success" or with a brief error message if it has failed. You can drill down to see the full error message in the case of any failures.


Community resources

Brian, a Community Grandmaster within the Matillion ETL Community, has performed a number of upgrades and migrations to Matillion ETL instances. He has collated considerations and suggestions pertinent to his experiences in this post.