Component Exports
  • Dark
    Light

Component Exports

  • Dark
    Light

Overview

All orchestration components and some transformation components support exporting runtime information into variables during job execution. Component export values are also used to populate the tasks view and the run history. The component's Export tab allows you to edit mappings between runtime information that the component makes available, and variables you have already defined.


List of export variables

This tables gives a list of common export variables. Some components will have additional component-specific variables.

Exported Value Type Description
Automatic Compression String Components such as Table Output allow data to be compressed if not already compressed. This variable indicates whether this compression occurred or not.
Completed At DateTime Time the component finished running in the job.
Component String Name of the component.
Duration Integer Length of time the component ran for.
Iteration Attempted Integer The number of iterations that this component attempts to reach (Max Iterations parameter).
Iteration Generated Integer The number of iterations that have been initiated. Iterators terminate after failure so this number will be the successful iterations plus any potential failure.
Iteration Successful Integer The number of iterations successfully performed. This is the max iteration number, minus failures and any unattempted iterations (since the component terminates after failure).
Message String Output message from component (usually empty).
Row Count Integer Number of rows the component has run over. Note: BigQuery doesn't always report row counts but attempting to retrieve the count won't cause an error.
Started At DateTime Time the component began running in the job.
Status String Status of the component.
Table Recreated String Will return No if the table already exists and the component is set to Create if not exists. In all other cases, will return Yes.

Usage

To export a value, you must map it to an existing job variable or environment variable.

  1. Click the component on the job canvas and click the Export tab.

  2. Click Edit to open the Export Variables dialog.

  3. Click + to create a new mapping, and click in the Source column of the newly added blank row to open the drop-down.

  4. Select the desired export variable from the drop-down.

  5. Click in the Target Variable column and select a target variable from the drop-down. This drop-down contains both environment variables and job variables.

  6. Click OK when you have completed all mappings.

Export Variables

Note
  • The type of the target variable (String, Integer, DateTime) must match the type of the source variable given in the table above.
  • Runtime information exported to job variables is only available while the job is running, and is no longer available after the job finishes running.

Grid variable export

When a job calls a child job with the Run Orchestration component, the values of grid variables can be exported from the child job to the parent job. This is managed from the component's Grid Export tab.

To export a grid variable between jobs:

  1. Define separate grid variables in both the child job and the parent job, as described in Creating grid variables. The variables don't have to have the same name, nor do the number or names of the columns in each variable have to be identical.
Note

The grid variable created in the child job must have its Behaviour set to Shared and its Visibility set to Public.

  1. Click the Run Orchestration component on the job canvas and click the Grid Export tab.

  2. The Grid Export tab lists the grid variables currently exported from that component. A component can contain multiple different grid variables, and each can be exported to a different grid in the parent job. To configure a new grid export, click +.

  3. In the Export Grid Variable dialog, select a grid variable from the Map Into drop-down. This is the grid in the parent job that the component's grid variable will be exported to. If you haven't previously created a suitable grid variable, you can click + to create a new one.

  4. Select a grid variable from the Grid Variable drop-down. This is the grid in the child job that will be exported.

  5. Under Column Mapping, each column in the parent grid is listed. From the drop-down next to each column, select the column in the child grid that will be exported to it. Every column in the parent grid must be matched with a column in the child grid, but you do not have to use every column in the child grid. You can also map the same column in the child grid to multiple columns in the parent grid if you wish. The data type of the mapped columns must be the same.

When the job runs and the Run Orchestration component is executed, the current values of the mapped grid columns will be exported to the columns of the grid variable in the parent job, over-writing any current value they may have.