Salesforce Incremental Load

Salesforce Incremental Load


This article is part of the series on Incremental Load Tools and Shared Jobs.

Overview

The Salesforce Incremental Load component lets users configure a pre-existing (read-only) Shared Job that will perform an incremental load from Salesforce. When the component is added to the Matillion ETL canvas, the Salesforce Incremental Load wizard is activated. Once this wizard is finalised, a unique component is added to the canvas, with a custom configuration provided by the user's setup choices.

In the Components panel, type "salesforce incremental load" to locate the component and add it to the canvas.

Find the component using the search bar


Salesforce Incremental Load Setup (Redshift)

1. Connection Details

Page 1 of the wizard gives users a brief explanation of the Salesforce Incremental Load Generator, and requires basic connection information.

Salesforce OAuth

Here, users can select a configured Salesforce OAuth entry from the dropdown list. To add a new OAuth entry, or edit or remove existing entries, click the Manage button and follow our Salesforce Query Authentication Guide to learn how to set up and authorise a Salesforce OAuth entry.

In this example an OAuth entry named "Salesforce" has been selected.


Parameter | Value

Next, specify any connection options. These exist as parameter=value pairs. To add a parameter, click . Users can consult the Salesforce Data Model for more information about connection options.


Sync Deleted Records

Check this box if you wish to synchronise record deletions from source data to the target table. By default, this box is not checked.

Click Next.

Connection Details


2. Data Sources

Page 2 of the wizard focuses on the data sources (tables) to load. If the OAuth entry and connection options were applied successfully, page 2 will show the success message.

Data Sources

Use the arrow buttons to select which data sources to add to the incremental load. Data sources in the left column are not set to be included; data sources in the right column will be included in the incremental load.

By default, the right column is empty, and users must manually add the columns they wish to load.

In this example, the data sources "Account" and "AcceptedEventRelation" have been selected for the incremental load.

Use the text fields above the columns to refine your searches.

Click Next.

Data Sources


3. Columns

Page 3 of the wizard requires users to confirm the columns to be loaded from each selected data source.

Select Columns

Click to navigate into a similar multiple select dialog that, but that is unique to the data source and its columns. From here, use the arrow buttons to add or remove any columns. By default, all columns are set to be loaded.

Click Next.

Columns


4. Configuration

Page 4 of the wizard requires users to specify data warehouse details.

Property Setting Description
Staging Bucket Select Select the S3 bucket from the dropdown list for data staging. The avaliable buckets depend on the selected Redshift cluster.
Staging Table Prefix String Specify a prefix to be added to all tables that are staged.
Stage Schema Select Select the Redshift schema via which tables will be staged.
Target Table Prefix String Specify a prefix to be added to all tables in the load.
Target Schema Select Select the Redshift schema into which tables will be loaded.
Target Distribution Style Select Select the distribution style.
  1. All Copy rows to all nodes in the Redshift Cluster.
  2. Even Distribute rows around the Redshift cluster evenly. This is the default setting.
Concurrency Select Select whether to load data in a concurrent or sequential method.

Click Create & Run to finish the setup; or else click Back to cycle back through the wizard.

Data warehouse configuration


Salesforce Incremental Load Setup (Snowflake)

1. Connection Details

Page 1 of the wizard gives users a brief explanation of the Salesforce Incremental Load Generator, and requires basic connection information.

Salesforce OAuth

Here, users can select a configured Salesforce OAuth entry from the dropdown list. To add a new OAuth entry, or edit or remove existing entries, click the Manage button and follow our Salesforce Query Authentication Guide to learn how to set up and authorise a Salesforce OAuth entry.

In this example an OAuth entry named "Salesforce" has been selected.


Parameter | Value

Next, specify any connection options. These exist as parameter=value pairs. To add a parameter, click . Users can consult the Salesforce Data Model for more information about connection options.


Sync Deleted Records

Check this box if you wish to synchronise record deletions from source data to the target table. By default, this box is not checked.

Click Next.

Connection Details


2. Data Sources

Page 2 of the wizard focuses on the data sources (tables) to load. If the OAuth entry and connection options were applied successfully, page 2 will show the success message.

Data Sources

Use the arrow buttons to select which data sources to add to the incremental load. Data sources in the left column are not set to be included; data sources in the right column will be included in the incremental load.

By default, the right column is empty, and users must manually add the columns they wish to load.

In this example, the data sources "Account" and "AcceptedEventRelation" have been selected for the incremental load.

Use the text fields above the columns to refine your searches.

Click Next.

Data Sources


3. Columns

Page 3 of the wizard requires users to confirm the columns to be loaded from each selected data source.

Select Columns

Click to navigate into a similar multiple select dialog that, but that is unique to the data source and its columns. From here, use the arrow buttons to add or remove any columns. By default, all columns are set to be loaded.

Click Next.

Columns


4. Staging Configuration

Page 4 of the wizard requires users to specify details for data staging.

Property Setting Description
Staging Table Prefix String Specify a prefix to be added to all tables that are staged.
Staging Warehouse Select Select the staging warehouse.
Staging Database Select Select the staging database.
Staging Schema Select Select the staging schema.

Click Next.

Staging configuration


5. Target Configuration

Page 5 of the wizard requires users to specify target data warehouse details.

Property Setting Description
Target Table Prefix String Specify a prefix to be added to all tables in the load.
Target Warehouse Select Select the target warehouse.
Target Database Select Select the target database.
Target Schema Select Select the target schema.
Concurrency Select Select whether to load data in a concurrent or sequential method.

Click Create & Run to finish the setup; or else click Back to cycle back through the wizard.

Target Configuraton


Salesforce Incremental Load Setup (BigQuery)

1. Connection Details

Page 1 of the wizard gives users a brief explanation of the Salesforce Incremental Load Generator, and requires basic connection information.

Salesforce OAuth

Here, users can select a configured Salesforce OAuth entry from the dropdown list. To add a new OAuth entry, or edit or remove existing entries, click the Manage button and follow our Salesforce Query Authentication Guide to learn how to set up and authorise a Salesforce OAuth entry.

In this example an OAuth entry named "Salesforce" has been selected.


Parameter | Value

Next, specify any connection options. These exist as parameter=value pairs. To add a parameter, click . Users can consult the Salesforce Data Model for more information about connection options.


Sync Deleted Records

Check this box if you wish to synchronise record deletions from source data to the target table. By default, this box is not checked.

Click Next.

Connection Details


2. Data Sources

Page 2 of the wizard focuses on the data sources (tables) to load. If the OAuth entry and connection options were applied successfully, page 2 will show the success message.

Data Sources

Use the arrow buttons to select which data sources to add to the incremental load. Data sources in the left column are not set to be included; data sources in the right column will be included in the incremental load.

By default, the right column is empty, and users must manually add the columns they wish to load.

In this example, the data sources "Account" and "AcceptedEventRelation" have been selected for the incremental load.

Use the text fields above the columns to refine your searches.

Click Next.

Data Sources


3. Columns

Page 3 of the wizard requires users to confirm the columns to be loaded from each selected data source.

Select Columns

Click to navigate into a similar multiple select dialog that, but that is unique to the data source and its columns. From here, use the arrow buttons to add or remove any columns. By default, all columns are set to be loaded.

Click Next.

Columns


4. Staging Configuration

Page 4 of the wizard requires users to specify details for data staging.

Property Setting Description
Cloud Storage Area Select Select the Cloud Storage bucket to stage the data.
Staging Table Prefix String Specify a prefix to be added to all tables that are staged.
Staging Project Select Select the staging project.
Staging Dataset Select Select the staging dataset.

Click Next.

Staging configuration


5. Target Configuration

Page 5 of the wizard requires users to specify target data warehouse details.

Property Setting Description
Target Table Prefix String Specify a prefix to be added to all tables in the load.
Target Project Select Select the target BigQuery Project.
Target Dataset Select Select the target dataset.
Concurrency Select Select whether to load data in a concurrent or sequential method.

Click Create & Run to finish the setup; or else click Back to cycle back through the wizard.

Target Configuraton


Completion

Upon completion of the wizard, a Salesforce Incremental Load component will be present on the canvas. This component is not identical to the Salesforce Query component, but it does utilise the same ideas to perform an incremental load from Salesforce.

To make changes to the component, simply click on it, and click into any of the component's properties—exactly as you would with any other Matillion ETL component. Users should avoid making changes to the component during a running workflow.


Blacklisted Tables

The following tables have been blacklisted:

  • "Announcement"
  • "ContentDocumentLink"
  • "ContentFolderItem"
  • "ContentFolderMember"
  • "ContentHubItem"
  • "DataStatistics"
  • "DatacloudAddress"
  • "EntityParticle"
  • "FeedAttachment"
  • "FeedItem"
  • "FeedRevision"
  • "FieldDefinition"
  • "FlexQueueItem"
  • "ListViewChartInstance"
  • "Note"
  • "OutgoingEmail"
  • "OutgoingEmailRelation"
  • "OwnerChangeOptionInfo"
  • "PicklistValueInfo"
  • "PlatformAction"
  • "RelationshipDomain"
  • "RelationshipInfo"
  • "SearchLayout"
  • "SiteDetail"
  • "TaskFeed"
  • "TaskRelation"
  • "UserEntityAccess"
  • "UserFieldAccess"
  • "Vote"