Matillion ETL — Upgrading to version 1.60 and above
  • Dark
    Light

Matillion ETL — Upgrading to version 1.60 and above

  • Dark
    Light

Overview

This page covers the technical details of how to upgrade and rollback between the Matillion ETL non-universal and universal packages along with an overview of the reasoning behind the project.



Background: consolidation of .rpm Packages

Prior to v1.60, Matillion ETL was published as three yum packages:

matillion-emerald
matillion-emerald-byol
matillion-emerald-cdata

Each installation of Matillion ETL would then have one combination of:

matillion-emerald AND matillion-emerald-cdata

Or:

matillion-emerald-byol AND matillion-emerald-cdata

Both of the matillion-emerald and matillion-emerald-byol packages then had both cloud platform and cloud data warehouse specific builds. For example:

emerald-1.46.10-centos7-aws-snowflake-platform.rpm   (matillion-emerald package)
emerald-1.59.9-centos7-azure-snowflake-platform.rpm  (matillion-emerald package)
emerald-1.51.9-centos7-aws-redshift-byol.rpm (matillion-emerald-byol package)

However, matillion-emerald-cdata was always published as a single universal build. For example:

emerald-cdata-1.50.12.rpm (matillion-emerald-cdata package)
emerald-cdata-1.56.10.rpm (matillion-emerald-cdata package)

As will be obvious, this large amount of builds/variation resulted in quite a complicated installation and complicated the tasks of building, validating, distributing, and providing support.



What's new: matillion-universal package

Therefore, as part of the v1.60 build, there have been two important changes:

  1. The matillion-emerald and matillion-emerald-byol packages have been consolidated to form the matillion-universal package.
  2. There are no longer cloud platform and cloud data warehouse specific builds for the matillion-universal package. Instead, there is one universally compatible build per Matillion Version.

For example:

emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)
emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)

From v1.60 onwards, each Matillion ETL installation will require:

matillion-universal *AND* matillion-emerald-cdata



Finding the matillion-universal packages

Two new yum repositories have been created, which can be found at the following URLs:

  1. matillion-universal ——— https://artifacts.matillion.com/rpm/matillion-metl/stable/
  2. matillion-emerald-cdata ——— https://artifacts.matillion.com/rpm/matillion-cdata/stable/

An example yum repo config for each is shown below:

Usually found at /etc/yum.repos.d/matillion-cdata.repo:

[matillion-cdata]
name=matillion-cdata
baseurl=https://artifacts.matillion.com/rpm/matillion-cdata/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-cdata/stable/matillion.key
gpgcheck=1
enabled=1


Usually found at /etc/yum.repos.d/matillion-universal.repo:

[matillion-universal]
name=matillion-universal
baseurl=https://artifacts.matillion.com/rpm/matillion-metl/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-metl/stable/matillion.key
gpgcheck=1
enabled=1


Please Note

If the machine(s) is/are running in an egress-restricted environment, please ensure artifacts.matillion.com is allow-listed within the domain/network.


Upgrading to the universal packages

Users have two available methods to upgrade to the universal packages (Matillion ETL v1.60 and beyond).

Please Note

Before performing any of these upgrades, please ensure relevant backups of root disks and managed databases are taken in case of the need to roll back. Read our documentation for more information on how to do this: Updating Matillion ETL.

Method 1: Updating/Upgrading via migration

To upgrade via migration, please launch a machine image of version 1.60 or higher and follow the steps here: Updating: Machine Images & Migration .

Method 2: Updating/Upgrading in-place (non-clustered environments)

Updating via the GUI is possible for non-clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package.

To do this, please follow the steps here: Updating: In-place updates.

Please Note

It is only possible to upgrade between the final version of the old packages (1.59.12) to the universal packages (>1.60.2).

In practice, if the installation is on a version below 1.59.12, the machine must be updated to 1.59.12 first, then the universal packages will be available (i.e. two upgrade hops).

Both these upgrades will be possible to do in the UI in series.

In the UI, the upgrade between 1.59.12 and >1.60 will only note the matillion-emerald-cdata package being available on the destination version. This is expected, and is due to the nature of the yum transaction.

Updating/Upgrading in-place (clustered environments)

Updating via the console is possible for clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package in place.

1. Ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:

service tomcat8 stop

2. To identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:

yum list installed | grep matillion-emerald

3. This will return either:

matillion-emerald-byol-<some version>

Or:

matillion-emerald-<some version>

4. To "swap" to the universal packages, the following command can be run:

yum swap <package from above> matillion-universal

5. For example, if swapping matillion-emerald-byol:

yum swap matillion-emerald-byol matillion-universal

Loaded plugins: fastestmirror, matillion-ensure-yumhook
Loading mirror speeds from cached hostfile
 * base: download.cf.centos.org
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: download.cf.centos.org
 * updates: download.cf.centos.org
Resolving Dependencies
--> Running transaction check
---> Package matillion-emerald-byol.noarch 0:1.59.12-1 will be erased
---> Package matillion-universal.noarch 0:1.60.2-1 will be installed
--> Processing Dependency: matillion-emerald-cdata = 1.60.2 for package: matillion-universal-1.60.2-1.noarch
--> Running transaction check
---> Package matillion-emerald-cdata.noarch 0:1.59.12-2 will be updated
---> Package matillion-emerald-cdata.noarch 0:1.60.2-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================
 PackageArch  VersionRepository  Size
==================================================================================================================================
Installing:
 matillion-universalnoarch1.60.2-1   matillion-universal351 M
Removing:
 matillion-emerald-byol noarch1.59.12-1  @matillion 431 M
Updating for dependencies:
 matillion-emerald-cdatanoarch1.60.2-1   matillion-cdata457 M

Transaction Summary
==================================================================================================================================
Install  1 Package
Upgrade ( 1 Dependent package)
Remove   1 Package

Total download size: 808 M
Is this ok [y/d/N]: y
Downloading packages:
...
Excluded for brevity
...  4/4 

Removed:
  matillion-emerald-byol.noarch 0:1.59.11-1   

Installed:
  matillion-universal.noarch 0:1.60.2-1   

Dependency Updated:
  matillion-emerald-cdata.noarch 0:1.60.2-1   

Complete!

6. Finally, restart the service once both nodes have been upgraded:

service tomcat8 start

Once the upgrade is confirmed to be successful, to prevent the old packages being installed alongside matillion-universal please run the command:

mv /etc/yum.repos.d/matillion.repo /etc/yum.repos.d/matillion.repo.bkp

This will de-index the repository file mirror for the old packages.



Ongoing updates on matillion-universal

Updates on 1.60 and beyond are then consumed by running the following command:

yum update matillion-universal

Not by running yum update matillion-*



Rollback

Rolling back via the console is possible for all environments. This will transfer the existing installation from matillion-universal back onto either the matillion-emerald or matillion-emerald-byol.

1. First, ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:

service tomcat8 stop

2. If previously the matillion.repo repository file was de-indexed, please re-index by running:

mv /etc/yum.repos.d/matillion.repo.bkp /etc/yum.repos.d/matillion.repo

This will re-index the repository file mirror for the old packages.

3. Now, to identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:

yum list available | grep matillion-emerald

This will return either:

matillion-emerald-byol-<some version>

Or:

matillion-emerald-<some version>

4. To "swap" back off the universal packages, the following command can be run:

yum swap matillion-universal <package from above>

5. For example, if swapping for matillion-emerald-byol:

yum swap matillion-universal matillion-emerald-byol

Transaction output:

    yum swap matillion-universal matillion-emerald-byol
    Resolving Dependencies
    --> Running transaction check
    ---> Package matillion-emerald-byol.noarch 0:1.59.12-1 will be installed
    ---> Package matillion-universal.noarch 0:1.60.2-1 will be erased
    --> Finished Dependency Resolution
    Dependencies Resolved
     ==================================================================================================================================
     Package   Arch  VersionRepository   Size
    ==================================================================================================================================
    Installing:
     matillion-emerald-byolnoarch1.59.12-1  matillion   351 M
    Removing:
     matillion-universal   noarch1.60.2-1   @matillion-universal 431 M
    Transaction Summary ==================================================================================================================================
    Install  1 Package
    Remove   1 Package
    
   Total download size: 351 M
   Is this ok [y/d/N]: y
   
   ...
   excluded for brevity
   ...
   Removed:
     matillion-universal.noarch 0:1.60.2-1   
    
   Installed:
      matillion-emerald-byol.noarch 0:1.59.12-1   
    
   Complete!
   

6. In the above transaction, the matillion-emerald-cdata package was not pushed down, so that needs to be downgraded to match the version of either the matillion-emerald or matillion-emerald-byol package. Therefore, run the command:

yum downgrade matillion-emerald-cdata-1.59.12

Example output:

yum downgrade matillion-emerald-cdata-1.59.12

Resolving Dependencies
--> Running transaction check
---> Package matillion-emerald-cdata.noarch 0:1.59.12-2 will be a downgrade
---> Package matillion-emerald-cdata.noarch 0:1.60.2-1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================
 Package   Arch Version Repository   Size
==================================================================================================================================
Downgrading:
 matillion-emerald-cdata   noarch   1.59.12-2   matillion   458 M

Transaction Summary
==================================================================================================================================
Downgrade  1 Package

Total download size: 458 M
Is this ok [y/d/N]: y
Downloading packages:
emerald-cdata-1.59.11.rpm  | 458 MB  00:00:18 
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : matillion-emerald-cdata-1.59.12-2.noarch   1/2 
  Cleanup: matillion-emerald-cdata-1.60.2-1.noarch2/2 
  Verifying  : matillion-emerald-cdata-1.59.12-2.noarch   1/2 
  Verifying  : matillion-emerald-cdata-1.60.2-1.noarch2/2 

Removed:
  matillion-emerald-cdata.noarch 0:1.60.2-1   

Installed:
  matillion-emerald-cdata.noarch 0:1.59.12-2  

Complete!

7. Finally, restart the service once both nodes have been downgraded:

service tomcat8 start