Server Migration Tool
Warning: The Server Migration Tool and this page are deprecated and exist to aid customers on older versions of Matillion ETL. The Server Migration Tool is now upgraded and integrated into the Matillion ETL client. See this article for more information.
Note: Your network administrator must ensure that the target Matillion ETL server allows incoming TCP/IP connections from the source Matillion ETL server.
The server migration tool allows users to push all assets from current server to new server. This tool runs as a shell script when placed on the server you wish to move. Through the tool, users are able to filter what is migrated, although everything will be migrated by default.
This can be useful in a number of cases:
- Setting up a new live environment by pushing all assets from a staging environment
- Setting up a test environment by pushing a single project from live to a test environment to diagnose
- Migrating a Single Node Matillion Setup to a Cluster by provisioning a new cluster using Cloudformation and then pushing everything to it from the existing environment.
To use this tool, simply copy the scripts to your server (the Matillion ETL instance), SSH onto the server and run the python script with the target hostname (including port if required) as an argument. It may be necessary to update your python installation before running the tool.
It is also required that API permissions are granted to the user for both source and target server.
The tool comes with help that can be accessed through the --help argument. A list of optional arguments is given below.
|--source-server SOURCE_SERVER||Source Server Hostname (Default: http://127.0.0.1)|
|--skip-drivers||Skip JDBC Driver migration (true/false)|
|--skip-profiles||Skip API Profile migration (true/false)|
|--skip-projects||Skip Matillion Project migration (true/false)|
|--skip-schedules||Skip Schedule migration (otherwise schedules are migrated but disabled) (true/false)|
|--skip-sqssetup||Skip SQS Listener setup (even when included queues are initially disabled) (true/false)|
|--skip-oauth||Skip OAuth Token migration (true/false)|
|--allow-deletes||Allow deleting same-named resources. Projects are deleted and recreated. (true/false)|
|--project-filter PROJECT_FILTER||Regular Expression to match against Group- name/Project-name/Version-name; default .* (migrate everything)|
Their usage is as below:
migrate_server.py [-h] [--source-server SOURCE_SERVER] [--skip-drivers][--skip-profiles] [--skip-projects][--skip-schedules] [--skip-sqssetup] [--skip-oauth][--allow-deletes] [--project-filter PROJECT_FILTER] target_server
Below is a generic example for running the migration tool from the EC2 Console with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).
Below is a generic example for running the migration tool from an SSH session with the necessary updates that may be required to run to script. Simply download migrate.py to /tmp and use the following 3 commands (replacing the source and target server IP with your own).
Note: If running the Migration Tool from the source or the target server then use your private-ip or 127.0.0.1:8080 to identify localhost instead of using the IP address. Servers may not recognise their public ip and may throw an error.
sudo yum install python34-pip.noarch
sudo pip-3.4 install requests
sudo /tmp/migrate_server.py --allow-deletes --source-server http://source-server-ip http://target-server-ip
- You will be asked to provide source credentials (ec2-user/instance-id) and target credentials (ec2-user/instance-id). The script will then proceed to migrate all available assets.
Note: Schedules on the new server will be disabled by default.
- Disable SQS integration on the old server and shutdown the server.
- On the new server, enable SQS Integration and Enable all schedules.