S3 Manifest Builder
  • Dark

S3 Manifest Builder

  • Dark

This article is specific to the following platforms - Redshift.

S3 Manifest Builder

Build and upload a manifest file which describes a set of files (S3 objects) to load based upon a pattern using a regular expression.

If the pattern is simply a prefix, this is not required as the S3 Load can use a prefix directly. However, for more complicated file-naming schemes, use this component to build a manifest file first, and then use an S3 Load with the manifest option and a path pointing to the generated manifest file.


Property Setting Description
Name Text A human-readable name for the component.
Source S3 Path S3 Tree The S3 bucket (and optionally the path within it) to search.
Manifest Path S3 Tree The S3 location to write the manifest file to.
Manifest File name Text The file name for the manifest.
Filter Regular Expression Regular Expression This is a regular expression. All names found from Source S3 Path are tested against this expression, and if they match they are included in the manifest.
This requires an exact match, so if you are searching for text anywhere within the name you will need to include wildcards before and after the text.
Mandatory Select Yes writes the Mandatory option into the manifest, meaning the subsequent S3 load will fail if the file is not present. No makes the file optional, and no error will be raised if the file is missing.

Variable Exports

This component makes the following values available to export into variables:

Source Description
Matched Objects The number of objects matching the filter pattern and written to the Manifest.

What's Next