The Cloud Pub/Sub component allows messages to be published to a topic on Google Cloud Platform (GCP). Other applications can subscribe to these topics to receive such messages. The end result is fast, reliable, and secure communication between GCP services applications.
To publish messages from Matillion ETL, you must first ensure that topics are available to publish to (and ideally, that relevant applications are subscribed to these topics). To view, edit, and create topics, visit the Topics GCP page while logged into your GCP account and project.
Official help for getting starting with Cloud Pub/Sub can be found here.
The below table cites the Cloud Pub/Sub component's configuration properties and any actions required by the user.
|Name||String||Input the descriptive name for the component.|
|Project||Select||Select the name of the GCP project to use. The chosen project must be affiliated with the GCP account connected to Matillion ETL.|
|Topic||Select||The name of the topic to publish to. Matillion ETL will automatically gather topics available to your project. To view and create topics, navigate to Pub/Sub→Topics from the main upper-left menu on the GCP dashboard or follow this link.|
|Message||String||Input the contents of the message.|
|Attributes||Key-value Pairs||Assign key-value pairs to be included with the message to be used by the subscribed application. These may commonly be data regarding the message itself.|
This example will demonstrate how to set up a topic on GCP, submit a message to it, and then check the message. To begin, ensure you are logged into the GCP Console and are on the relevant project, then visit the Cloud Pub/Sub Topic list.
Click Create Topic and input a name when prompted - we have called our example topic 'docspub'.
Select your new topic from the list. From your topic's page, click Create Subscription and give the Subscription a name - we have called our example Subscription 'docsub'. Ensure the Delivery Type is set to "Pull" before clicking Create to complete the Subscription.
Now, over in Matillion ETL, we have created an Orchestration Job with the Cloud Pub/Sub component.
The component is set up as shown below. We enter the project name (the one used while creating the topic) and the topic name we created earlier. Now, we can add some data to the message - in this case, a short piece of text we can use to verify our message has been delivered correctly.
Next, we run our job, and Matillion ETL sends the messsage to the topic. To pull data from the topic, we need to use the Cloud SDK tool as described here.
When on the Cloud SDK, we use the following line to check our subscription:
gcloud beta pubsub subscriptions docsub
As can be seen, the message is now pulled from the topic and printed to the screen, confirming its success.