Loop over values of a simple sequence.
This component implements a simple loop over rows of fixed data values. It enables you to run an attached component multiple times, each time with different values set on any number of variables. Those variable can be referenced from the attached component.
To attach the iterator to another component, use the blue output connector and link to the desired component. To detach, right-click on the attached component and click Disconnect from Iterator.
If you need to iterate more than one component, put them into a separate orchestration job or transformation job and use a Run Transformation or Run Orchestration component attached to the iterator. In this way, you can run an entire ETL flow multiple times, once for each row of variable values.
All iterator components are limited to a maximum 5000 iterations.
|Name||String||A human-readable name for the component.|
|Concurrency||Select||Concurrent: Iterations are run concurrently. This requires all "Variables to Iterate" to be defined as copied variables, so that each iteration gets its own copy of the variable isolated from the same variable being used by other concurrent executions.
Sequential: Iterations are done in sequence, waiting for each to complete before starting the next. This is the default setting.
Note: The maximum concurrency is limited by the number of available threads (2x the number of processors on your cloud instance).
|Variable to Iterate||Select||Choose an existing variable to iterate. Only numeric variables are available.|
|Iteration Values||Variable 1||Enter as many rows as you require. Each row becomes an iteration.|
|Variable n||For each variable chosen in Variables to Iterate, provide a value for that iteration.|
|Break on Failure||Select||No: Attempt to run the attached component for each iteration, regardless of success or failure. This is the default setting.
Yes: If the attached component does not run successfully, fail immediately.
Note i: If a failure occurs during any iteration, the failure link is followed. This parameter controls whether it is followed immediately or after all iterations have been attempted.
Note ii: This property is only available when Concurrency is set to Sequential. When set to Concurrent, all iterations will be attempted.
|Record Values In Task History||Select||Choose whether to record iteration values in the Matillion ETL Task History. The default setting is Yes.|
|Stop On Condition||Select||Select Yes to stop the iteration based on a condition specified in the Condition property. The default setting is No.
For this property to be available, set Concurrency to Sequential.
|Mode||Select||Select the method of creating the condition.
Simple: A no-code Condition UI will open, where users must specify an Input Variable, Qualifier, Comparator, and Value using drop-down menus and text fields. This is the default setting.
Advanced: An editor will open, where users must write the condition manually using SQL.
|Condition (Simple mode)||Input Variable||An input variable to form a condition around.|
|Qualifier||Is: Compares the input variable to the value using the comparator.
Not: Reverses the effect of the comparison, so "Equals" becomes "Not equals", "Less than" becomes "Greater than or equal to", etc.
|Comparator||Select the comparator. Available comparison operators include "Less than", "Less than or equal to", "Equal to", "Greater than or equal to", "Greater than", and "Blank".|
|Value||Specify the value to be compared.|
|Combine Conditions||Select||Use the defined conditions in combination with one another according to either And or Or.
This property is only available when Mode is set to Simple.
This component makes the following values available to export into variables:
|Iteration Attempted||The number of iterations that this component attempts to reach (Max Iterations parameter).|
|Iteration Generated||The number of iterations that have been initiated. Iterators terminate after failure so this number will be the successful iterations plus any potential failure.|
|Iteration Successful||The number of iterations successfully performed. This is the max iteration number, minus failures and any unattempted iterations (since the component terminates after failure).|
This example iterates through two different airlines, and two different US States. For each of those 4 combinations, a transformation job is run. The variables are available to use in that transformation job to filter the incoming data.
We need to define the variable we intend to iterate, and provide some sensible default values.
In the configuration for the Iterator, the two variables are selected and each combination of EI (Aer Lingus), BA (British Airways) carriers, and CA (California) and PA (Pennsylvania) states are entered.
When the transformation runs, the AnalyseFlightData transformation job is executed 4 times, each time setting the carrier_code and state variables.
An alternative configuration would be to nest two Fixed Iterators together, iterating the values for carrier_code in the first, and state in the second.