Matillion ETL for Synapse Release Notes

Matillion ETL for Synapse Release Notes


We highly recommend reading the Updating Matillion ETL article for the safest way to update your instance.

Matillion ETL for Azure Synapse Analytics v1.50.8

  • Fixed an issue where the Task History API filter would always interpret times as +12 hours from the given value
  • Added independent authentication checks when users attempt to:
    • Import or update user permissions or groups
    • Get a list of external security users
    • Restart the seryeaver
    • Load SSL Configuration
    • Save Security Configuration
    • Upload certificates
    • Upload Drivers

Matillion ETL for Azure Synapse Analytics v1.50.7


(01/12/2020)

  • Fixed an issue with Git Integration where Git commit and Git push actions were taking an unexpectedly long time to complete in Matillion ETL instances with a high number of Matillion "versions".
  • (Microsoft Azure-hosted instances only) added an improvement to how Matillion ETL behaves with Azure Queue Listener's GetMessageRead transactions. The number of transactions is now limited to three per minute.

Matillion ETL for Azure Synapse Analytics v1.50.6

(12/11/2020)

Note: Due to a required upgrade of the Jira API driver, this component may not function correctly for users updating HA clusters. See this tech note for more information.

New Features

  • New Intercom Output component allows users to push their data from Synapse to Intercom
  • Parity components
  • Bearer Token added as an available authentication method to API Query Component
  • Security updates:
    • A new “Change My Password” dialog in the project menu allows users to change their own password for the Matillion ETL instance
    • New Global Access permissions group that allows admins to give a user full access to all projects.
    • Admins can assign an initial permission group to new users, meaning that users can now start with restricted access rather than needing to log in once before an admin can limit their access.


Improvements

  • Objects in the Recycle bin now show when they were sent to the Recycle Bin
  • Manage API Profiles is now Manage Query Profiles and has received a significant UI update to its wizard
    • Two new pagination strategies are now supported; Cursor Based and Page based. 
    • The wizard now supports bearer token as a distinct authentication type
    • The authentication tab has had some UI improvements 
    • Users can now switch between summary mode and advanced mode
    • You can now find a Configurable Review screen at the end of the Query Profile wizard


Bug Fixes and Driver Updates

  • Fixed an error where connectors would fail when using an existing S3 location to stage data.
  • Square Driver Updated (v19.0.7270.0 to v20.07538.0)
  • Google Leads Manager Driver Updated (v19.0.7367.0 to v20.0.7488.0)
  • Google Sheets Driver Updated (v19.0.7285.0 to v20.0.7550.0)
  • Salesforce Driver version Reverted (v20.0.7587.0 to v19.0.7332). This circumvents a bug in the newer driver that often resulted in Text column sizes returning 0
  • Shopify API Driver Updated (v19.0.7354 to v19.0.7485)
  • Jira API Driver Updated (v19.0.7326 to v20.0.7564)
  • Netsuite API Driver Updated (v19.0.7156 to v20.0.7513)



Matillion ETL for Azure Synapse Analytics v1.48.6

  • Fixed an issue where the Jira Query component was unable to query system tables following a recent driver update.

Matillion ETL for Synapse v1.48

New Features

  • Webhook Post component gives users a simplified approach to posting notifications to multiple unique URLs in order to make other users aware of job behaviors. “Manage Webhook Payloads” allows users to predefine payloads for later use.
  • New Salesforce Output component allows users to push data from their data warehouse to a Salesforce Object of their choice.
  • New Create External Table allows users to utilise External Tables to reference and query data direct from storage rather than in the data warehouse.
  • New Split Fields component splits table fields based on user-determined criteria
  • New Unpivot component component can be used to convert table columns into additional table rows, the opposite of our existing Pivot component.
  • New Table Output allows users to write their transformation job data flows into a table.

Improvements

  • Create Your Own Connector process improvements
    • Import / Export for connector profiles is now possible through the UI and API
    • Server Migration function now supports migrating extract profiles
    • New pagination strategies: Offset Based and Link-header based
  • You can now encrypt user-defined credentials using the Azure Key Vault
  • Query Result to Grid component now supports WHERE clauses
  • Assert components can now auto-populate semi-structured data and persist metadata values in Assert View

Notable Bug Fixes

  • Pivot Component does not quote resulting columns when columns do not conform to the "standard" naming convention
  • Flatten Variant component - Double Quotes around “Property” values in the Mapping Columns property does not return results
  • S3 Load Generator gives an error reading Header when instructed to ignore reading headers
  • JDBC Incremental Load always performs a full load



Matillion ETL for Azure Synapse Analytics v1.47.9

  • Exported Query Profiles now correctly obfuscate credentials.


Matillion ETL for Azure Synapse Analytics v1.47.8

  • Fixed an issue where Shopify Query was not fetching all data correctly.

Matillion ETL for Azure Synapse Analytics v1.47.6

  • Fixed an issue where the Matillion ETL client would get stuck loading projects.
  • Fixed an issue with the Postgres driver: "The SSLSocketFactory class provided org.postgresql.ssl.DefaultJavaFactory could not be instantiated".
  • Fixed an issue where the JDBC properties file could not longer be edited in 1.47.



Matillion ETL for Azure Synapse Analytics v1.47

Note: Due to some technical issues with this release, we do not recommend customers manually update to this version. We highly recommend users on 1.47 to update to 1.47.6 immediately.

New Features

  • New Detect Changes transformation component allows comparison of two similar tables to yield a new column that details whether rows have been inserted, deleted or changed.
  • New Pivot transformation component can pivot data from a selected column into multiple columns, partitioned by user-defined criteria.
  • New Generate Sequence transformation component allows users to create rows whose data are increments of a user-defined numerical sequence.
  • New Multi-table input transformation component reads data from multiple tables according to their names matching a user-defined pattern.
  • Azure Blob Load Generator helps generate compatible "Create Table" and "S3 Load" components by sampling delimited data files on Azure Blob Storage and guessing the layout.
  • OpenID Connect support for third party login providers:
    • You can now configure Matillion ETL to authenticate with any Open ID Connect provider.
    • Default support for Google, Microsoft and Okta plus a “Generic” option.

Improvements

  • Assert components now auto-populate their properties when connected to another component. For example: Connecting Assert Table to an S3 Load component will automatically configure it to use the same table as the S3 Load component.
  • Rewrite Table now allows Partitioning of data by a field.
  • The Environment Panel now has right-click context menu for related options


Matillion ETL for Azure Synapse Analytics v1.46.8

  • Added support for Azure Blob Storage when running Azure Government Cloud
  • Fixed an issue in the Matillion ETL v1 API where the flag "onConflict=OVERWRITE" during job import was not working.
  • Fixed an issue where running a Transformation job more than once concurrently can cause it to interfere with itself.
  • Multiple bug fixes for "Assert" Components
    • Multi-line data types now supported
    • Fixed an issue where mismatched value types pass validation
    • No longer validates on size and precision fields where not relevant
    • Multiple fixes to error messages
    • Fixed an error when asserting values on a view containing a datetime column

Matillion ETL for Azure Synapse Analytics v1.46

New Components

  • New Zendesk Talk Query component allows users to bring their Zendesk Talk data into a table.
  • New Python Script component allows users to write custom Python scripts that are executed on the instance to perform a wide variety of tasks .
  • New “Assert” components can be used to test an existing object (Table, External Table, Scalar Variable or View) against user-defined criteria. This allows users to prevent jobs continuing if resources are not in their expected state.
  • New “Rank” Transformation component allows users to find the numerical rank of values in a column and output these ranks to a new column.
  • New “First/Last” Transformation component allows users to create groups of data and return only the first or last rows from that group.
  • New “Lead/Lag” Transformation component allows users to determine values of a new column based on data preceding or following that source row with a given offset.

New Features

  • New Recycle Bin feature gives users the opportunity to recover any deleted jobs before committing to delete them completely.

Improvements

  • Reworked Component panel organization gives an alphabetized list of all connector components.
  • Connectors that only import data into external tables have been renamed as “Extract” components.
  • New functions have been added to the Window Calculation component:
    • First/Last
    • Lead/Lag
    • Rank