Data Transfer Component

Data Transfer Component



Data Transfer

The Data Transfer component enables users to transfer files from a chosen source to a chosen target.

This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a Source Type and a Target Type; the component's other properties will change to reflect those choices.

Currently supported data sources: Azure Blob Storage, Box, Dropbox, FTP, Google Cloud Storage, HDFS, HTTP, HTTPS, S3 Bucket, SFTP, Windows Fileshare.

Currently supported targets: Azure Blob Storage, Google Cloud Storage, HDFS, S3 Bucket, SFTP, Windows Fileshare.

Note: FTPS is not supported through this component. We recommend using SFTP instead, or installing a tool that supports FTPS and calling it from a Bash Script component. You could also do this using cURL, which is available as standard.

When reading from or writing from Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.

Below we list the Base Properties for this component that are independent of Source and Target Types. Then, property tables are given for Source and Target properties grouped by their Source and Target Types.



Redshift Properties

Property Setting Description
Name String Input descriptive name for the component.
Source Type Select Select the type of data source.
Unpack ZIP File Yes/No Select if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type Select Select the target type for the new file.
Target Object Name String The filename of the new file.
Gzip Data Yes/No Select if you wish to Gzip the transfered data when it arrives at the target.

Source Properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The URL, including full path and file name, that points to the source file that exists on Azure Blob Storage.
FTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Google Cloud Storage Source URL Text The URL, including full path and file name, that points to the source file.
HDFS Source URL Text The URL, including full path and file name, that points to the source file.
HTTP Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
HTTPS Perform Certificate Validation Choice Check the SSL certificate for the host is valid before taking data.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Microsoft Sharepoint URL Text The URL is the web address that you visit to sign into your SharePoint account.
User Text A valid SharePoint username to use for authentication.
Password Text A valid SharePoint password. Users have the option to store their password inside the component; however, we highly recommend using the Password Manager feature instead.
SharePoint Edition Select Select your edition of SharePoint.
Connection Options Parameter/Value Parameter: A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version. They are usually not required, since sensible defaults are assumed.
Value: A value for the given parameter.
File Type Select Choose whether the file type is a document or attachment.
Library Select The location where you can upload, create, update, and collaborate on files. For more information, see here.
File URL Text The URL that points to the file.
S3 Source URL Text The URL, including full path and file name, that points to the source file.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Source SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Source URL Text The URL, including full path and file name, that points to the source file.
Source Domain Text The domain that the source file is located on.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Target properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The full URL that points to the target file location on Azure Blob Storage.
Google Cloud Storage Target URL Text The URL (without file name) that points to where the new file will be created.
HDFS Target URL Text The URL (without file name) that points to where the new file will be created.
S3 Target URL S3 Tree The URL of the S3 bucket to get the files from. This follows the format s3://bucket-name/path .
Access Control List Options Select Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
Encryption Select Decide on how the files are to be encrypted inside the target S3 Bucket.
None: No encryption.
SSE Encryption: Encrypt the data according to a key stored on KMS.
S3 Encryption: Encrypt the data according to a key stored on an S3 bucket.
KMS Key ID Select The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Target URL Text The URL (without file name) that points to where the new file will be created.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Target SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Target URL Text The URL (without file name) that points to where the new file will be created.
Target Domain Text The domain that the newly created file is to be located on.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Snowflake Properties

Property Setting Description
Name String Input descriptive name for the component.
Source Type Select Select the type of data source.
Unpack ZIP File Yes/No Select if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type Select Select the target type for the new file.
Target Object Name String The filename of the new file.
Gzip Data Yes/No Select if you wish to Gzip the transfered data when it arrives at the target.

Source Properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The URL, including full path and file name, that points to the source file that exists on Azure Blob Storage.
FTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Google Cloud Storage Source URL Text The URL, including full path and file name, that points to the source file.
HDFS Source URL Text The URL, including full path and file name, that points to the source file.
HTTP Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
HTTPS Perform Certificate Validation Choice Check the SSL certificate for the host is valid before taking data.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Microsoft Sharepoint URL Text The URL is the web address that you visit to sign into your SharePoint account.
User Text A valid SharePoint username to use for authentication.
Password Text A valid SharePoint password. Users have the option to store their password inside the component; however, we highly recommend using the Password Manager feature instead.
SharePoint Edition Select Select your edition of SharePoint.
Connection Options Parameter/Value Parameter: A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version. They are usually not required, since sensible defaults are assumed.
Value: A value for the given parameter.
File Type Select Choose whether the file type is a document or attachment.
Library Select The location where you can upload, create, update, and collaborate on files. For more information, see here.
File URL Text The URL that points to the file.
S3 Source URL Text The URL, including full path and file name, that points to the source file.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Source SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Source URL Text The URL, including full path and file name, that points to the source file.
Source Domain Text The domain that the source file is located on.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Target properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The full URL that points to the target file location on Azure Blob Storage.
Google Cloud Storage Target URL Text The URL (without file name) that points to where the new file will be created.
HDFS Target URL Text The URL (without file name) that points to where the new file will be created.
S3 Target URL S3 Tree The URL of the S3 bucket to get the files from. This follows the format s3://bucket-name/path .
Access Control List Options Select Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
Encryption Select Decide on how the files are to be encrypted inside the target S3 Bucket.
None: No encryption.
SSE Encryption: Encrypt the data according to a key stored on KMS.
S3 Encryption: Encrypt the data according to a key stored on an S3 bucket.
KMS Key ID Select (AWS Only) The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Target URL Text The URL (without file name) that points to where the new file will be created.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Target SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Target URL Text The URL (without file name) that points to where the new file will be created.
Target Domain Text The domain that the newly created file is to be located on.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

BigQuery Properties

Property Setting Description
Name String Input descriptive name for the component.
Source Type Select Select the type of data source.
Unpack ZIP File Yes/No Select if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type Select Select the target type for the new file.
Target Object Name String The filename of the new file.
Gzip Data Yes/No Select if you wish to Gzip the transfered data when it arrives at the target.

Source Properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The URL, including full path and file name, that points to the source file that exists on Azure Blob Storage.
FTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Google Cloud Storage Source URL Text The URL, including full path and file name, that points to the source file.
HDFS Source URL Text The URL, including full path and file name, that points to the source file.
HTTP Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
HTTPS Perform Certificate Validation Choice Check the SSL certificate for the host is valid before taking data.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Microsoft Sharepoint URL Text The URL is the web address that you visit to sign into your SharePoint account.
User Text A valid SharePoint username to use for authentication.
Password Text A valid SharePoint password. Users have the option to store their password inside the component; however, we highly recommend using the Password Manager feature instead.
SharePoint Edition Select Select your edition of SharePoint.
Connection Options Parameter/Value Parameter: A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version. They are usually not required, since sensible defaults are assumed.
Value: A value for the given parameter.
File Type Select Choose whether the file type is a document or attachment.
Library Select The location where you can upload, create, update, and collaborate on files. For more information, see here.
File URL Text The URL that points to the file.
S3 Source URL Text The URL, including full path and file name, that points to the source file.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Source SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Source URL Text The URL, including full path and file name, that points to the source file.
Source Domain Text The domain that the source file is located on.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Target properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The full URL that points to the target file location on Azure Blob Storage.
Google Cloud Storage Target URL Text The URL (without file name) that points to where the new file will be created.
HDFS Target URL Text The URL (without file name) that points to where the new file will be created.
S3 Target URL S3 Tree The URL of the S3 bucket to get the files from. This follows the format s3://bucket-name/path .
Access Control List Options Select Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
Encryption Select Decide on how the files are to be encrypted inside the target S3 Bucket.
None: No encryption.
SSE Encryption: Encrypt the data according to a key stored on KMS.
S3 Encryption: Encrypt the data according to a key stored on an S3 bucket.
KMS Key ID Select The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Target URL Text The URL (without file name) that points to where the new file will be created.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Target SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Target URL Text The URL (without file name) that points to where the new file will be created.
Target Domain Text The domain that the newly created file is to be located on.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Synapse Properties

Property Setting Description
Name String Input descriptive name for the component.
Source Type Select Select the type of data source.
Unpack ZIP File Yes/No Select if the source data is a ZIP file that you wish to unpack before being transferred.
Target Type Select Select the target type for the new file.
Target Object Name String The filename of the new file.
Gzip Data Yes/No Select if you wish to Gzip the transfered data when it arrives at the target.

Source Properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The URL, including full path and file name, that points to the source file that exists on Azure Blob Storage.
FTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Google Cloud Storage Source URL Text The URL, including full path and file name, that points to the source file.
HDFS Source URL Text The URL, including full path and file name, that points to the source file.
HTTP Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
HTTPS Perform Certificate Validation Choice Check the SSL certificate for the host is valid before taking data.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Microsoft Sharepoint URL Text The URL is the web address that you visit to sign into your SharePoint account.
User Text A valid SharePoint username to use for authentication.
Password Text A valid SharePoint password. Users have the option to store their password inside the component; however, we highly recommend using the Password Manager feature instead.
SharePoint Edition Select Select your edition of SharePoint.
Connection Options Parameter/Value Parameter: A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version. They are usually not required, since sensible defaults are assumed.
Value: A value for the given parameter.
File Type Select Choose whether the file type is a document or attachment.
Library Select The location where you can upload, create, update, and collaborate on files. For more information, see here.
File URL Text The URL that points to the file.
S3 Source URL Text The URL, including full path and file name, that points to the source file.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Source URL Text The URL, including full path and file name, that points to the source file.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Source SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Source URL Text The URL, including full path and file name, that points to the source file.
Source Domain Text The domain that the source file is located on.
Source Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Source Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Target properties
Source Property Setting Description
Azure Blob Storage Blob Location Text The full URL that points to the target file location on Azure Blob Storage.
Google Cloud Storage Target URL Text The URL (without file name) that points to where the new file will be created.
HDFS Target URL Text The URL (without file name) that points to where the new file will be created.
S3 Target URL S3 Tree The URL of the S3 bucket to get the files from. This follows the format s3://bucket-name/path .
Access Control List Options Select Choose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
Encryption Select Decide on how the files are to be encrypted inside the target S3 Bucket.
None: No encryption.
SSE Encryption: Encrypt the data according to a key stored on KMS.
S3 Encryption: Encrypt the data according to a key stored on an S3 bucket.
KMS Key ID Select The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
SFTP Set Home Directory as Root Yes/No Yes: URLs are relative to the user's home directory.
No: URLs are relative to the server root.
Target URL Text The URL (without file name) that points to where the new file will be created.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.
Target SFTP Key Text This is your SFTP Private Key. It is optional and will only be used if the data source requests it.
This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
Windows Fileshare Target URL Text The URL (without file name) that points to where the new file will be created.
Target Domain Text The domain that the newly created file is to be located on.
Target Username Text This is your URL connection username. It is optional and will only be used if the data source requests it.
Target Password Text This is your URL connection password. It is optional and will only be used if the data source requests it.

Copying Files to an Azure Premium Storage blob

When copying files to an Azure Premium Storage blob, Matillion may provide the following error:

Self-suppression not permitted.

This is because, unlike standard Azure Storage, Azure Premium Storage does not support block blobs, append blobs, files, tables, or queues. Premium Storage supports only page blobs that are incrementally sized.

A page blob is a collection of 512-byte pages that are optimised for random read and write operations. Thus, all writes must be 512-byte aligned and so any file that is not sized a multiple of 512 will fail to write.

For additional information about Azure Storage blobs, we recommend consulting the Microsoft Azure documentation.