Connecting to Google Cloud Storage

Overview

Matillion CDC can load data from any of your pipelines into Google Cloud Storage. Follow the steps on this page to configure Google Cloud Storage as a destination.

Please refer to the following topics to help you set up CDC for Google Cloud Storage:

Google Cloud Storage prerequisites

You need a Google Cloud Platform account with administrative permissions to create resources. Signing up is free.

Permissions to create and manage IAM Roles on Google Cloud Storage buckets. Users must be able to create a bucket (if one doesn't already exist), add/modify bucket policies, and upload files to the bucket.

The Google Service account used by the Agent container should have the following permissions for the Google Cloud Storage bucket.

storage.buckets.get

storage.objects.get

storage.multipartUploads.abort

storage.multipartUploads.create

storage.multipartUploads.list

storage.multipartUploads.listParts

storage.objects.create

storage.objects.delete

storage.objects.get

storage.objects.list

storage.objects.update

Provide a unique prefix name for each pipeline.

Note

Read Google's Cloud Storage documentation to learn more about the service.

Connect to Google Cloud Storage

Select destination

After you configure the source during a CDC pipeline creation, you will be directed to choose a destination you would like to load your data into.

In the Choose destination page, select Google Cloud Storage.

Configure Google Cloud Storage Connection Settings

Specify the following settings in the Connect to Google Cloud Storage destination page:

Field Name	Description
Bucket	This refers to the name of the Google Cloud Storage bucket you want to use as a destination. Find your bucket name in the Google Cloud Management Console.
Prefix	Prefix is the name of the 'folder' or a location within the storage bucket that all CDC data for this pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes.

Field Name

Description

Bucket

This refers to the name of the Google Cloud Storage bucket you want to use as a destination. Find your bucket name in the Google Cloud Management Console.

Prefix

Prefix is the name of the 'folder' or a location within the storage bucket that all CDC data for this pipeline should be saved to. You could have multiple agents using the same bucket with different prefixes.

Note

A pipeline must have one unique prefix.

Test Connection

You can test your connection by using the Test connection button at the bottom; if the test is successful, click Test and Continue.