.png){kind=logo}
Create or Refresh External Table shared job
-
DarkLight
Create or Refresh External Table shared job
-
DarkLight
This shared job is specific to the following platform - Snowflake.
Overview
This shared job maintains two external tables, used by the Sync Single Table shared job. The first table, named '<External Table>' contains all the change events from a given CDC pipeline. The second table, named '<External Table>_VERSION_METADATA' contains the metadata for each version of the source tables.
On first execution, the job will create the external tables over the Snowflake stage. The tables will be created with partition columns to enable efficient querying for new data.
If the external tables already exist, they will be refreshed to contain any new files.
Properties
Snowflake Properties | ||
|---|---|---|
| Property | Description | |
| Target Database | The Snowflake database containing the external table with change events and the target table. | |
| Target Schema | The Snowflake schema containing the external table with change events and the target table. | |
| Snowflake External Stage | The name of the existing external stage that contains the files output by the CDC pipeline. | |
| External Table | The external table in Snowflake containing the CDC change events. | |
| Cloud Storage URL | The URL of the location where the CDC change event files are stored. This should have the format:s3://<bucket>/<prefix>/or azure://<account>.blob.core.windows.net/<container>/<prefix>/ |
|