Schema Drift Support
Several pipelines include Schema Drift support. Schema Drift is what occurs when changes are made the source data or API settings, thereby creating a difference between the source and your configuration in the pipeline. If these changes are not handled appropriately, the pipeline can fail or process incorrect information. As such, pipelines that have Schema Drift support will handle changes to the source data and continue operating. You may still need to make changes if you notice schema drift occuring. Schema Drift support handles the following changes to source data:
- Missing columns. Missing columns are loaded as NULL.
- Data type changes. See below text for more detail.
- Missing tables. Missing tables are not loaded. All other tables will continue to be loaded. You will need to update your pipeline settings to remove the missing tables.
- Newly added tables and fields. The pipeline does not automatically add new tables/fields to the data source to your pipeline. The pipeline is not otherwise affected, and you can manually add new tables and fields to your pipeline as required.
Data Type changes are accommodated, but if these are not compatible changes for the target cloud platform, the current column will be renamed as <column_name>_datetime and the column re-purposed as the new datatype. The format of the datetime extension is
_yyyymmddhhmmss , e.g.
_20210113110334 and will be the same for all columns in the same table in the same Pipeline execution. The new column will be NULL up to the date of change - this should be considered for downstream dependencies such as views, reports, etc.