GCP Account Setup for BigQuery and Storage

GCP Account Setup for BigQuery and Storage


Matillion ETL for BigQuery requires users to set up their GCP account with BigQuery and GCP authentication in the Matillion ETL instance itself.


To begin, browse to the Google Cloud Platform Console site at https://console.cloud.google.com/.

If you do not currently have a project set up, create a new one from the Project menu.

The newly-created project can now be selected.

In this example, we have a project named 'Getting Started'.

Through the upper-left dropdown menu, browse to the 'IAM & Admin' area of the dashboard.

At the left-hand menu, browse to 'Service Accounts'. A list of current service accounts will be present on this page and new service accounts can be added using the 'Create Service Account' button at the top.

Service accounts can be named as you please but must 'Furnish a new private key' via the checkbox. Ensure this key is in JSON format using the appropriate radio button. 

In addition to this, the service account must be given a set of roles that allow Matillion to function.
Important: Your GCP account must have a valid billing account set up in order to view all roles.

The required roles are:

Heading Role
Project Editor
BigQuery BigQuery Admin
BigQuery Data Editor
BigQuery Data Owner
BigQuery Data Viewer
BigQuery User
Storage Storage Admin
Storage Object Admin
Storage Object Creator
Storage Object Viewer

Ensure each of the above roles are selected before clicking 'Create' to make the new Service Account. Upon creation, the key associated with this Service Account should immediately begin downloading. If not, a new key can be generated using the menu beside the desired Service Account. Again, ensure the downloaded key is in JSON format.

At this point, it is worth noting your Project's ID. While still on your desired project, use the top-left menu to return to 'Home' and take note of the Project ID.

You will also require a Dataset to use. Again from the top-left menu, browse to 'BigQuery'. On the left-side, public Datasets available to you are listed. Above that, the project's private Datasets are also listed. 

To add a new Dataset, use the small dropdown menu and select 'Create Dataset'.

Make note of the Dataset ID and click OK.

Now that your GCP account has BigQuery and Storage enabled, you are ready to Launch Matillion ETL for BigQuery.