API v1 - Queue

API v1 - Queue


Overview

The queue endpoint allows Queues to be managed externally, similarly to how they are managed within the Matillion ETL client via Project Manage SQS Configuration or Manage Pub/Sub Configuration. Queue configuration can then be exported, imported and tested using the API v1

Important Information

  • This document is part of a series on API Profiles and the Matillion ETL v1 API
  • This process requires the Matillion ETL instance URL, and the username and password of a user with appropraite permissions.

Queue API Endpoints

Queue API is available on standard REST-based APIs that uses HTTP or HTTPS request to GET and POST data. The Queue API service is accessed through the Uniform Resource Identifier (URI). All following references in this document will assume the API Base URL has been specified. The available API endpoints are listed below:

Function Method Path URL
Export Queue configurations GET /export https://<InstanceURI>/rest/v1/queue/export
Import Queue configurations POST /import https://<InstanceURI>/rest/v1/queue/import
Test Queue configurations GET /test https://<InstanceURI>/rest/v1/queue/test

Please Note

  • The username and password of a user with access to the Queue within Matillion ETL instance will need to be used to authenticate the connection. These may need to be specified before the command when using a command line tool—for example: -u user:password.
  • When exporting or importing Queue configurations, the source file or content type and target file will also need to be specified.
  • Matillion ETL can currently only use Queue configurations formatted in JSON.

Error

While manually editing an imported Queue configuration file in JSON is possible, it is not recommended.


API Parameters and Description

Below is the list of endpoint parameters and their brief description:

Parameter Description
<InstanceURI> The server IP address or domain name of the Matillion ETL instance
<export> This allows the API to export the Queue configuration to another Matillion ETL instance
<import> This allows the API to import the Queue configuration from another Matillion ETL instance
<test> This allows the API to test the validity of the Queue configuration settings

Endpoints and Server Response

These APIs provide a REST-based web service—offering ease of use and a flexible choice of programming language—and can be used to access and analyse data and service provider content. Below are a number of example endpoints along with the corresponding summary, parameters and server responses:

GET/export

This endpoint permits the API to export the Queue configurations from the Matillion ETL instance.

  • Base URL
    https://<InstanceURI>/rest/v1/queue/export
  • Server Response
    {
          "credentialsName": "Instance Credentials",
          "successEnabled": true,
          "compressSuccess": false,
          "failureEnabled": true,
          "compressFailure": false,
          "region": "eu-west-1",
          "sqsEnabled": true,
          "listenQueue": "LISTEN_QUEUE",
          "successQueue": "SUCCESS_QUEUE",
          "failureQueue": "FAIL_QUEUE"
        }

POST/import

This endpoint permits the API to import Queue configurations into the Matillion ETL instance. This will be a "POST" method API call as it will have to attach the project, in the body as a JSON file to import into the Matillion ETL instance.

  • Base URL
    https://<InstanceURI>/rest/v1/queue/import
  • POST Body (JSON)
    {
          "credentialsName": "Instance Credentials",
          "successEnabled": true,
          "compressSuccess": false,
          "failureEnabled": true,
          "compressFailure": false,
          "region": "eu-west-1",
          "sqsEnabled": true,
          "listenQueue": "LISTEN_QUEUE",
          "successQueue": "SUCCESS_QUEUE",
          "failureQueue": "FAIL_QUEUE"
        }

    Below is the description of the fields included in the POST body:

    Field name Description
    credentialsName The credentials used to talk to the Queues
    successEnabled Enable this to place a message on a Queue when an Orchestration Job has completed
    compressSuccess The body of the "completed" message on the Queue will be compressed
    failureEnabled Enable this to place a message on a Queue when an Orchestration Job has failed
    compressFailure The body of the "failed" message on the Queue will be compressed
    region The region in which the Queue exists
    sqsEnabled Enable this to turn on the sub-system that listens to Queues
    listenQueue The name of the Queue to listen for messages
    successQueue The name of the Queue on which "completed" messages will be placed
    failQueue The name of the Queue on which "failed" messages will be placed
  • Server Response
    {
          "success": true,
          "msg": "Successfully imported queue.",
          "id": 0
        }

GET/test

This endpoint permits the API to test the Queue configurations on the Matillion ETL instance.

  • Base URL
    https://<InstanceURI>/rest/v1/queue/test
  • Server Response
    {
            "success": true,
            "msg": "Queue test was successful.",
            "id": 0
          }