Contents x
RDS Bulk Output
  • Dark
    Light
Contents

RDS Bulk Output

  • Dark
    Light
This article is specific to the following platforms - Snowflake - Redshift.

RDS Bulk Output

Matillion ETL for Snowflake: this feature is only available for instances hosted on AWS.

RDS Bulk Output lets users load the contents of a table (or view) into a table in an Amazon RDS database.


Properties

Snowflake Properties

Property Setting Description
Name Text A human-readable name for the component.
RDS Type Select Select the database. Available options include:
aurora: Amazon Aurora (see the AWS Documentation).
mariadb: MariaDB (see the AWS Documentation).
mysql: MySQL (see the AWS Documentation).
postgresql: PostgreSQL (see the AWS Documentation).
SQL Server (Microsoft Driver): Microsoft SQL Server (see the AWS Documentation).
RDS Endpoint Text This is the RDS Database Endpoint. If the IAM role attached to the instance (or the manually entered credentials associated with the current environment) has the permissions granted to query the RDS endpoints, you may select the RDS endpoint from a list. Otherwise, you must enter it manually - it can be found in the RDS console and is a long dotted-name and port number, separated by a colon.
Database Name Text This is the name of the database within your RDS instance.
Username Text This is your RDS connection username.
Password Text This is your RDS connection password. The password is masked, so it can be set, but not read.Users have the option to store their password inside the component, but we highly recommend using the Password Manager option.
JDBC Options Parameter A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version.
They are usually not required as sensible defaults are assumed.
Value The parameter value.
Database Text The name of the database you wish to source data from.
Schema Select Select the table schema. The special value, [Environment Default] will use the schema defined in the environment. For more information on using multiple schemas, see this article.
Source Table Select The table (or view) on your cluster to copy to RDS.
Target Table Text Provide a new table name.
Load Columns Select Choose the columns to load into RDS. This parameter can be empty, which means that all columns will be loaded.
Table Maintenance Select None - assume the RDS database already has the table defined with the correct structure.
Create if not exists - Only create the table if it doesn't already exist.
Replace - Always drop and re-create the table. Use with care!
Primary Key Select Specify a set of columns to be used as the primary key for the target table. This is optional, but if specified allows you to UPSERT existing data in the target table.
Truncate Target Table Select Whether or not to truncate the target table before loading data.
On Warnings Select Choose whether to Continue with the load if an error is raised or to Fail the run.
Additional Copy Options Text Any additional options that you want to apply to the copy. Some of these may conflict with the options the component already sets—in particular, care is taken to escape the data to ensure it loads into the target database even if the data contains row and/or column delimiters, so you should never override the escape or delimiter options.
Options are documented here:
Batch Size Integer This is optional, and specifies the number of rows to load to the target between each COMMIT. On a very large export, this may be desirable to keep the size of the Amazon RDS log files from growing very large before the data is committed.

Redshift Properties

Property Setting Description
Name Text A human-readable name for the component.
RDS Type Select Select the database. Available options include:
aurora: Amazon Aurora (see the AWS Documentation).
mariadb: MariaDB (see the AWS Documentation).
mysql: MySQL (see the AWS Documentation).
postgresql: PostgreSQL (see the AWS Documentation).
SQL Server (Microsoft Driver): Microsoft SQL Server (see the AWS Documentation).
RDS Endpoint Text This is the RDS Database Endpoint. If the IAM role attached to the instance (or the manually entered credentials associated with the current environment) has the permissions granted to query the RDS endpoints, you may select the RDS endpoint from a list. Otherwise, you must enter it manually - it can be found in the RDS console and is a long dotted-name and port number, separated by a colon.
Database Name Text This is the name of the database within your RDS instance.
Username Text This is your RDS connection username.
Password Text This is your RDS connection password. The password is masked, so it can be set, but not read.Users have the option to store their password inside the component, but we highly recommend using the Password Manager option.
JDBC Options Parameter A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version.
They are usually not required as sensible defaults are assumed.
Value The parameter value.
Schema Select Select the table schema. The special value, [Environment Default] will use the schema defined in the environment. For more information on using multiple schemas, see this article.
Source Table Select The table (or view) on your cluster to copy to RDS.
Target Table Text Provide a new table name.
Load Columns Select Choose the columns to load into RDS. This parameter can be empty, which means that all columns will be loaded.
Table Maintenance Select None - assume the RDS database already has the table defined with the correct structure.
Create if not exists - Only create the table if it doesn't already exist.
Replace - Always drop and re-create the table. Use with care!
Primary Key Select Specify a set of columns to be used as the primary key for the target table. This is optional, but if specified allows you to UPSERT existing data in the target table.
Update strategy Select Replace - Existing rows with the same Primary Key values will be replaced.
Ignore - Existing rows with the same Primary Key values will be ignored.
Truncate Target Table Select Whether or not to truncate the target table before loading data.
On Warnings Select Choose whether to Continue with the load if an error is raised or to Fail the run.
Additional Copy Options Text Any additional options that you want to apply to the copy. Some of these may conflict with the options the component already sets—in particular, care is taken to escape the data to ensure it loads into the target database even if the data contains row and/or column delimiters, so you should never override the escape or delimiter options.
Options are documented here:
Batch Size Integer This is optional, and specifies the number of rows to load to the target between each COMMIT. On a very large export, this may be desirable to keep the size of the Amazon RDS log files from growing very large before the data is committed.

Strategy

A select query is issued against the source table. The output is formatted in an appropriate way to load into the target database, and data is streamed in.

Video

What's Next

Cookie consent

We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content based on your browsing and navigation history.