SCM Integration

SCM Integration


Overview

For some customers the best practice topology is to have multiple instances of Matillion ETL (e.g. for Development, Test and Live) and then manage the migration using a source control management system combined with a build server or script environment. This allows Matillion ETL to support DevOps scenarios where all configuration is held centrally.



Below, we outline a project called “Marketplace Data Warehouse” in a group called “Matillion”. The job is called “Load Marketplace Data” and exists in the version “BW_21-7-2016”. The following steps will guide you through a typical application of SCM to Matillion ETL.

Note: The following steps are generic and can be adapted for just about any source control management system. For the sake of example, we outline the below with respect to GIT, however videos for other SCM systems are given at the bottom of the page.
 

Part 1 - Export a project and commit to source control


Exporting all the data for a given version is simple. Identify the version to export, and ensure all the jobs are included in the result with export=true. (The URL is quoted to protect the & characters being interpreted by bash.)

curl -o <filename> -X GET -u <user>:<password> -H "Content-Type: application/json" "https://server-name-or-ip/rest/v0/projects?groupName=<groupname>&projectName=<Project Name>&versionName=<Version Name>&export=true"


e.g.

curl -o MatillionMarketplace.json -X GET -u <user>:<password> -H "Content-Type: application/json" "http://10.12.1.28/rest/v0/projects?groupName=Matillion&projectName=Marketplace%20Data%20Warehouse&versionName=BW_21-7-2016&export=true"


If you want to output formatted Json you can use the jq command. e.g.

curl -X GET -u <user>:<password> -H "Content-Type: application/json" "http://10.12.1.28/rest/v0/projects?groupName=Matillion&projectName=Marketplace%20Data%20Warehouse&versionName=BW_21-7-2016&export=true" | jq ‘.’ > MatillionMarketplace.json

Part 2 - To commit to GIT



To begin with GIT, create a workspace:

mkdir MatillionWorkspace

Change into the directory:

cd MatillionWorkspace

Initialise the git repository:

git init

Create a branch to match the version in Matillion ETL:

git checkout -b BW_21-7-2016

Add the files retrieved from the API:

git add *

Commit the files:

git commit -m "Matillion Marketplace initial commit"

Push changes to the repository:

git remote add origin https://matillion-admin@bitbucket.org/matillion/matillion-etl-scm.git
git push origin --all


Part 3 - Checkout a project from source control and push to a new Matillion ETL instance



Continuing from part 2, here we will be checking-out the Matillion ETL project version we checked-in earlier.

Check out the project from SCM. For example:

git clone https://matillion-admin@bitbucket.org/matillion/matillion-etl-scm.git

Change into the project:

cd matillion-etl-scm/

Ensure we are looking at the correct branch (or version):

git branch

To push the checked-out JSON data into a new server:

curl -X POST http://server-name-or-ip/rest/v0/projects -H "Content-Type: application/json" --data-binary @JsonFile.json

e.g.

curl -X -u <user>:<password> POST http://10.12.2.229/rest/v0/projects -H "Content-Type: application/json" --data-binary @MatillionMarketplace.json

The return value shows that the operation was a success. An invalid ID is returned from this call, since this operation generates many new ID’s (for the project, environment, version, jobs…) - you can see the current project structure by doing a GET request to the same URL but without the export=true option.

{"success":true,"msg":"Import successful","id":-1}

Videos

Github

 
 

Mercurial

 
 

Subversion

 
 

TFS

 

Github

 
 

Mercurial

 
 

Subversion

 
 

TFS

 

Github

 
 

Mercurial

 
 

Subversion

 
 

TFS