Agent (GKE) Kubernetes Template
It's possible to use a Kubernetes platform to manage CDC agent containers. The provided Kubernetes pod templates will allow the agent image to be retrieved from GKE(Kubernetes) and deployed to any compatible x86 Linux Kubernetes platform.
See the official documentation on Kubernetes pods for more information.
Using the Matillion CDC Agent in a Kubernetes platform is among the more manual methods of setting up a Matillion Data Loader CDC pipeline, with a great burden of knowledge on the user. This should only be attempted by users who are adept with Kubernetes and Google Cloud.
The template provides a blueprint for installation that you may use verbatim, but you may need to modify it to suit your own needs and rules governing your cloud infrastructure.
This template is intended for users who are accustomed to setting up their own Kubernetes platforms.
- A Google Cloud account with administrative permissions to create resources.
- A Google Kubernetes Engine cluster with Workload identity enabled.
- A Google Cloud Project ID where resources will be created.
- A Google Cloud Region.
- A Google Cloud Zone.
- A Google Cloud Storage bucket where the Matillion CDC Agent will land the data.
- A Google Service account to be used by the Pod via Workload identity. See the documentation.
- A Google service account with access to the Secret Manager and Cloud Storage bucket.
- A Google Secret Manager secret.
- Platform Key (Secret Manager), Secret name (Platform Key gets generated on the first agent creation. All the subsequent agents will use the same key).
- Database password (Secret Manager) Secret name.
- Google custom IAM role with the following permissions:
orgpolicy.policy.get resourcemanager.projects.get secretmanager.versions.access storage.buckets.get storage.multipartUploads.abort storage.multipartUploads.create storage.multipartUploads.list storage.multipartUploads.listParts storage.objects.create storage.objects.delete storage.objects.get storage.objects.list storage.objects.update
- The customer private cloud network and associated firewall rules should be configured so that the agent can communicate with the Matillion CDC platform.
- Access to the Matillion Hub account and Matillion Data Loader.
- CDC agent environment variables (generated in Matillion Data Loader when creating a new agent) Agent ID, Organization ID, and the Platform Websocket Endpoint URL.
- Matillion Data Loader platform key (generated once per Matillion Data Loader account the first time you make an agent).
- Google Cloud account with the ability to create an instance on a billable account and create/grant IAM roles. You may require an administrator from your organization to either give access or perform this process with you.
- Create a new namespace named
matillion-cdcin Kubernetes for CDC resources.
- Use a templated installation if at all possible.
- Consult your cloud/network administrator for advice on customer private cloud, Kubernetes Services, and Ingress.
- Use Workload identity to allow access to the Secret Manager secrets and Cloud Storage bucket.
The template needs the following environment variables for Matillion Data Loader to recognize the agent.
|ID_ORGANIZATION||This is provided to you by the Matillion Data Loader client when setting up a new agent.|
|ID_AGENT||This is provided to you by the Matillion Data Loader client when setting up a new agent.|
|PLATFORM_KEY_NAME||The name of the key storing your platform secret that is generated the first time you attempt to create an agent.|
|PLATFORM_KEY_PROVIDER||The service that supplies your platform key. This must be
|PLATFORM_WEBSOCKET_ENDPOINT||This value must be set to
|SECRET_PROVIDERS||The service that holds your database passwords. This must be
|GCP_PROJECTID||This is your Google Cloud Project ID.|