Agent (GCE) Template
This article details how to install the Matillion CDC Agent on Google Compute Engine using templates.
It's expected that users who choose to use a Terraform template will have a working knowledge of infrastructure as Code using Terraform in Google Cloud. Users should familiarize themselves with the official documentation before continuing:
- GCP Official documentation for using Terraform.
- HashiCorp official documentation for using Terraform on Google Cloud.
Terraform template files can be found in the Downloads section of this article.
The template provides a blueprint for installation that you may use verbatim, but you may need to modify it to suit your own needs and rules governing your cloud infrastructure.
This template will create the following resources in your Google Cloud account:
- A Google Compute Engine instance.
- A Google service account (to be used by the Compute Engine instance).
- A custom IAM role with permissions required to access the Cloud Storage bucket and Secret Manager.
- Grants service account the custom IAM roles to access the Cloud Storage bucket.
- Grants service account the custom IAM roles to access platform key secret and database password secret.
- Grants service account the
- Grants service account the
Edit the template
Users should inspect the template in a text editor and ensure all the values are specified before proceeding. In particular, the Agent ID, Organisation ID, and the Platform Websocket Endpoint URL environment variable should be edited to match the expected endpoint and region. See Environment Variables for more information.
The template assumes you have certain resources already set up in your GCP stack. You will also need to provide details on these resources, such as names and paths.
- Google Cloud Project ID where resources will be created.
- Google Cloud Region and Google Cloud Zone.
- Google Cloud Storage bucket name (the template doesn't create the Cloud Storage bucket).
- Google Secret Manager Secrets.
- Platform Key secret name (for information on the Platform Key see here.
- Database Password secret name.
- customer private cloud Network name to be attached to the Compute Engine instance.
- The customer private cloud network and associated firewall rules should be configured so that the agent can communicate with the Matillion CDC platform.
You as a user are also expected to have access to certain details and permissions:
- Access to a valid Terraform installation.
- Access to the Matillion Hub account and Matillion Data Loader.
- CDC Agent environment variables (generated in Matillion Data Loader when creating a new agent) Agent ID, Organisation ID, and the Platform Websocket Endpoint URL.
- Matillion Data Loader platform key (generated once per Matillion Data Loader account the first time you make an agent).
- Google Cloud account with the ability to create an instance on a billable account and create/grant IAM Roles. You may require an administrator from your organization to either give access or perform this process with you.
This template is intended for users who are accustomed to setting up their own GCE instance with a level of permissions and access to the resources.
- Create a new project specifically for CDC resources.
- Use an installation template, if possible.
- Consult your cloud/network administrator for advice on customer private clouds, subnets, and other resources such as Google Cloud regions.
- Keep resources in the same Google Cloud region.
The template needs the following environment variables in order for Matillion Data Loader to recognise the agent.
|project_id||This is your Google Cloud Project ID.|
|region||Google Cloud region where the Compute Engine instance will be deployed.|
|zone||Google Cloud zone where the Compute Engine instance will be deployed.|
|network_name||Google Cloud network to attach to the Compute Engine instance|
|instance_name||Name of the Compute Engine instance. For example,
|storage_bucket_name||Name of the Google Cloud Storage bucket where the agent will land the data.|
|organization_id||This is provided to you by the Matillion Data Loader client when setting up a new agent.|
|agent_id||This is provided to you by the Matillion Data Loader client when setting up a new agent.|
|platform_websocket_endpoint||Set this to
|platform_key_secret_name||Name of the Platform Key Secret stored in the Google Secret Manager.|
|database_password_secret_name||Name of the source Database Password Secret stored in the Google Secret Manager.|