Tables created

Overview

The ETL process creates some transient staging tables during a pipeline run that are deleted after a run, regardless of success.

Tables are also created permanently to support the pipeline process, including the target table that contains your source data.

Snowflake Properties
Table Type	Duration	Purpose
Target Table	Lifetime of a pipeline	Stores the flattened data from the agent for your source.
Metadata Table	Lifetime of a pipeline	Create/Append to a metadata table (named <target_table>_VERSION_METADATA) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
External Table	Lifetime of a pipeline	A Snowflake external table (named by the user as per the shared job parameter of the same name). This table is a view of all files in cloud storage for the pipeline.

Redshift Properties
Table Type	Duration	Purpose
Target Table	Lifetime of a pipeline	Stores the flattened data from the agent for your source.
Metadata Table	Lifetime of a pipeline	Create/Append to a metadata table (named <target_table>_version_metadata) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
Stage Table	Temporary - for a pipeline run	A temporary stage table (named <target_table>_<uuid>). Copies in all the relevant files for all the versions, which are cleaned up after each run.

Delta Lake Properties
Table Type	Duration	Purpose
Target Table	Lifetime of a pipeline	Stores the flattened data from the agent for your source.
Metadata Table	Lifetime of a pipeline	Create/Append to a metadata table (named <target_table>_version_metadata) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
Stage Tables (N)	Lifetime of a pipeline	Stage Tables (named <stage-prefix>, <source_database>, <source_schema>, <source_table>, <source_version>. These are external tables that provide a view of all files in cloud storage for the pipeline for a given version.

BigQuery Properties
Table Type	Duration	Purpose
Target Table	Lifetime of a pipeline	A native table. Stores the flattened data from the agent for your source. The table is named: <target_prefix><source_database>_<source_schema>_<source_table>
Metadata Table	Lifetime of a pipeline	A BigQuery external table. Create/Append to a metadata table which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required. The table is named: <stage_prefix><source_database>_<source_schema>_<source_table>_<version_metadata>
External Table	Lifetime of a pipeline	A BigQuery external table. This table is a view of all files in cloud storage for the pipeline. The table is named: <stage_prefix><source_database>_<source_schema>_<source_table>_<source_version>

Table of contents