Matillion ETL for Snowflake Release Notes
Important: Users with instances hosted on Microsoft Azure or Amazon Web Services should take notice of a security advisory relating to Matillion ETL v1.44 instances.
Attention: If you started your Matillion ETL instance at 1.42.x or earlier and have manually upgraded the Tomcat package to a newer version than 8.5.42 please be aware that upgrading may cause problems.If your Tomcat version has not been changed this should not affect you, although we would still recommend starting a new instance from the Marketplace to ensure no future upgrade issues.
We highly recommend reading the Updating Matillion ETL article for the safest way to update your instance. Do not hesitate to contact Matillion support for assistance in this matter.
Matillion ETL for Snowflake v1.46.10
Fixed an issue that caused intermittent hanging of transfgormation jobs following the 1.46.8 update.
Matillion ETL for Snowflake v1.46.8
- Added support for Azure Blob Storage when running Azure Government Cloud
- Fixed an issue in the Matillion ETL v1 API where the flag "onConflict=OVERWRITE" during job import was not working.
- Fixed an issue where running a Transformation job more than once concurrently can cause it to interfere with itself.
- Multiple bug fixes for "Assert" Components
- Multi-line data types now supported
- Fixed an issue where mismatched value types pass validation
- No longer validates on size and precision fields where not relevant
- Multiple fixes to error messages
- Fixed an error when asserting values on a view containing a datetime column
Matillion ETL for Snowflake v1.46
- New Zendesk Chat Extract component allows users to bring their Zendesk Chat data into an external table.
- New Zendesk Talk Query component allows users to bring their Zendesk Talk data into a table.
- New Salesforce Output component allows users to push data from their data warehouse to a Salesforce Object of their choice.
- New Unpivot component can be used to convert table columns into additional table rows, the opposite of our existing Pivot component.
- New MindSphere Extract component allows users to pull data from their MindSphere account into a table.
- New “Assert” components can be used to test an existing object (Table, External Table, Scalar Variable or View) against user-defined criteria. This allows users to prevent jobs continuing if resources are not in their expected state.
- New Recycle Bin feature gives users the opportunity to recover any deleted jobs before committing to delete them completely.
- Additional Data Sources added to the Zuora Query component
- Revenue Schedule
- Revenue Event
- Revenue Event Item
- Revenue Event Item Invoice Item
- Revenue Event Item Invoice Item Adjustment
- Revenue Charge Summary Item
- Revenue Charge Summary
- Reworked Component panel organization gives an alphabetized list of all connectors.
- Connectors that import raw data directly into external tables have been renamed as “Extract” components.
- A number of components have had property updates to better align with Snowflake.
- Updated Salesforce Query driver
Matillion ETL for Snowflake 1.44.16
Fixed an issue where a certain Matillion ETL functions such as Migrate, Python Script and COPY INTO commands may cause instance credentials to be stored in log files.
Matillion ETL for Snowflake 1.44.14
- Updated Google Ad Manager Query component from API v201905 to API v201908, anticipating the deprecation of v201905 in June 2020. Users can override this version change using the "Version" connection option with a value of "v201905".
Matillion ETL for Snowflake 1.44.13
- Fixed an issue where Shared Jobs are not correctly running concurrent connections and transformation jobs.
Matillion ETL for Snowflake 1.44.11
- Users experiencing issues pulling Ticket_Metrics columns in the Zendesk Query component while using a custom RSD config can see this article for potential solutions.
- Project Admin assignment option added to External Security users.
- Added new data sources for Zendesk Query component: group_memberships, groups, satisfaction_ratings, sla_policies, tags, ticket_audits, ticket_comments, ticket_fields, ticket_forms, ticket_metrics_events_incremental.
- Fixed an issue where Data Transfer component would fail to maintain a connection when loading from Sharepoint.
- Fixed an issue where Zendesk Query component was bringing in erroneously-empty date fields.
- Fixed an issue where Twitter Query component was attempting to use an outdated version of the AdStats data source.
- Fixed "Error parsing int value [null]" error from Google Ad Manager Query component.
- Non-functional 'FromFriendlyName' columns have been removed from Gmail Query component data sources.
Matillion ETL for Snowflake 1.44.10
- Reverted a change that affected column names generated by Extract Nested Data component which caused issues where jobs containing the Extract Nested Data component would break if created before 1.44.
Matillion ETL for Snowflake 1.44.9
- Fixed 'black screen' error caused by XSS filter being enabled by default.
Matillion ETL for Snowflake 1.44.8
- Job Start Control allows you to ‘prevent duplicate job’ runs by selecting the option in the scheduler. When selected this allows users to specify that a scheduled job run should not be queued if that same job from a previous schedule is still running, preventing duplicates from stacking up in the queue.
- Box and Dropbox source options added to Data Transfer Component, allowing users to move their stored files between cloud services.
- Amplitude connector allows users to connect to their Amplitude account and load their data into external tables.
- Box connector allows users to collect data regarding their stored files and load it into external tables.
- Dropbox connector allows users to collect data regarding their stored files and load it into external tables.
- Add New Data Sources added to the Google Analytics Query component allows users a way of adding new tables to the "Data Sources" property without the need to deal with custom schemas.
- Git Credentials can now be stored so users don’t need to re-enter credentials every time they want to fetch or push a branch.
- Snowflake private key “passphrase” protection is now available when connecting to Snowflake, offering users an additional layer of security.
- Project Admin option now added to users defined in Internal Security that allows these users to see and access all projects regardless of the project privacy settings.
Matillion ETL for Snowflake 1.42.11
- Upgrade: Added OAuth 2.0 support for Xero component (OAuth 1.0 now deprecated)
- Upgrade: Updated Xero Driver
- Fix: Customer seeing Database Query validation errors after updating from 1.39.8 to 1.42.x
- Fix: Create Table displaying errors when given a Snowflake Database Name containing a hyphen
Matillion ETL for Snowflake 1.42.10
- Upgraded BigQuery driver to v19
- Fix: SQS Queue erroneously disconnecting
- Fix: postgreS error: "could not open relation with OID"
- Fix: Intermittent Job Failures on Validation
Matillion ETL for Snowflake 1.42.8
- Fixed various issues and bugs including:
- Google OAuth creation to failing.
- Query To Grid component would sometimes use the wrong database name in advance mode.
- Instance performance would deteriorate over time when jobs run concurrently.
- Jobs with many transformation jobs that should run in parallel were sometimes not doing so.
Matillion ETL for Snowflake 1.42
- New API endpoints for Git Integrations in Matillion ETL. Use the Matillion ETL v1 API to initialise and clone repositories, switch branches, pull and push changes.
- Sage Intacct Query Component allows users to connect to their Sage Intacct data.
- Materialized and Secure Views now available from the Create View component
- S3 Unload component now allows JSON file formats to nest columns
- Azure Unload component now allows JSON file formats to next columns
- S3 Unload component now supports the Parquet file type
- SAPHana support added to the Database Query Component
- NetSuite Query component now supports OAuth. Read our NetSuite Query Authentication Guide.
- SharePoint Query component now supports OAuth. Read our SharePoint Query Authentication Guide
- Additional Improvements
- Snowflake Driver updated to 3.9.2
- Support added for MySQL driver v8
Matillion ETL for Snowflake 1.41.8
- Pardot connector lets users connect to their Salesforce Pardot data.
- Fixed an issue where Git Push to remote repository fails citing an 'Invalid Private Key'.
- Fixed an issue where the Snowflake driver would cause an Out Of Memory error.
Matillion ETL for Snowflake 1.41.5
- (AWS Hosted Instances Only) Fixed error when upgrading to a t3.xlarge instance that would cause schedules to run and unable to be stopped.
- Increased possible Column Name lengths for Grid Variable mappings.
Matillion ETL for Snowflake 1.41
- Instagram Component connects to your Instagram account and pulls data into Redshift.
- New Query Result To Scalar Component lets users write SQL queries to return scalar variables that can be used directly in Matillion ETL workflows.
- Go beyond a single connection to a target database with the new ‘Concurrent Connections’ option for faster concurrent loading (Enterprise Only)
- Data Lineage allows you to understand the effect that your complex transformation jobs will have on your data. Track a column backwards to its source to determine where and how calculations are applied. (Enterprise Feature).
- Additional Improvements
- The Shared Jobs now allows creators to mark parameters as ‘required’ so users of the job must provide input for these parameters.
- LDAP Configuration (External Security) now can be tested before confirming the configuration. The ability to run a test login is also available before committing changes.
- Recursive Validation means that jobs linked to others via Run Orchestration and Run Transformation will be validated alongside their parent job.
Matillion ETL for Snowflake 1.40.11
- Twitter driver upgraded to support Ads API v5
- DoubleClick For Publishers (Google Ads Manager) driver upgraded to support API v201811 and later.
- Windows Fileshare SMB2/3 Support for Data Transfer component and File Iterator component .
- SAP Netweaver Query - update to the location of additional files including a "JCo" connection library and the Java wrapper "sapjco3.jar".
- Fixed an issue where SAP Netweaver Query component was not correctly finding associated libraries.
- Fixed an issue where Hubspot OAuth registration link would fail.
Matillion ETL for Snowflake 1.40 (Features Video)
- New data staging components for:
- Support for Structured Data
- New Extract Nested Data Component allows users to unpack their nested data structures (such as JSONs) guided by a visual tree structure of the data.
- Support for Snowflake Streams:
- Keypair Authentication now supported for Snowflake environment credentials.
- Git functionality allows users to work with local and remote repositories, managing Project Versions as commits and branches seamlessly inside the Matillion ETL client. (Enterprise Only)
- User Interface
- Changes to error and validation messaging in product.
- Improve plus(+) sign and grid visibility.
- UI Refresh - Tabs with no component selected.
- Keyboard Shortcut Ctrl +A for single-line text input fields.
- Order the Component Tree.
- Add Manage Variables button in Code Editors.
- Filter added to Sample Tab.
- Simple JSON or XML parser for Configure API Profile Dialog.
- All sorting of Project versions in Manage Versions.
- Add more JDBC Table Metadata To Grid Fields.
- Dynamics CRM Query component now supports CRM2011accesible from CRM Version dropdown.
- JDBC Incremental Load - Prepopulate URL Field with Connection Prefix.
- Create File Format Component has new properties available.
- The API Profile Generator component is now able to find files in the current directory the user is in, rather than specifying the whole path of the file.
- Additional Improvements
- Remove row count from Result.java.
- AMIs upgraded to Python 3.6.
- Warning: This upgrade may prevent previously-installed Python modules from working. We recommend users install pip3.6 and then reinstall modules.
- Azure nc and jq packages are pre-installed.
Matillion ETL for Snowflake 1.39 (Features Video)
- (AWS-Only) Users can now make use of Create External Table and Refresh External Table components.
- New Data Staging components to quickly and easily bring your data into Snowflake:
- S3 Load Generator tool interface has been reworked for a smoother experience.
- (Azure Only) Azure Queue Message Configuration allows users to set their Matillion ETL instance to listen to and create Azure Queue Storage messages that can be used to automate workflows. This matches the SQS functionality already available to Matillion ETL users on AWS.
- (Azure Only) Azure Blob Load Generator helps generate compatible "Create Table" and "S3 Load" components by sampling delimited data files on Azure Blob Storage and guessing the layout.
- Alter Warehouse now has an option for setting the Pivot Component rotates table data
- Lead/Lag Component now has the option of ignoring Null values.
- The Matillion ETL API (v1) now includes an endpoint that allows creating Versions.
- Database Query and RDS Query components now have Basic Mode similar to other Data Staging components that offers a codeless alternative to using these services.
- Additional endpoints available for Zuora Query and Zuora Bulk Query components.
Matillion ETL for Snowflake 1.38.8
- Updated Magento Query driver.
- Updated Google Analytics driver.
- New method of single-account authentication for Stripe Query.
- Improved handling of cluster synchronisation
- Reworked logged diagnostic information.
- Fixed an error where jobs could hang indefinitely due to a problem with the handling of temporary files.
- (Azure) Caching of API calls to Azure Storage resources to alleviate issues with users hitting the Azure API rate limit.
Matillion ETL for Snowflake 1.38 (Features Video)
- CDC based on DMS gives Snowflake on AWS users the power to automatically keep their Snowflake tables in-sync with their source database tables using Matillion.
- Product Improvement Metrics can now be gathered within Matillion ETL. When you update your instance of Matillion ETL for Snowflake you will be prompted to share anonymized Product Improvement Metrics data. Matillion will use this information to build you a better product. Absolutely no personal data is collected through this service.
- Create Project now has a multi-step wizard configuration complete with tooltips to help guide users through creating a Matillion ETL for Snowflake Project.
- Configuring Environments now uses a multi-step wizard configuration complete with tooltips to help guide users through creating and editing Environments.
- Reintroducing a dialog option that allows “Text Mode” configuration of Variables (namely, Manage Environment Variables and Manage Job Variables). Use the Text Mode to enter or copy variables data in a plain text format.
- A new Sharepoint Query component allows users to connect to their Sharepoint account.
- The new Azure Queue Storage Message component allows users to easily post messages to the Azure Queue Storage service from within Matillion ETL for Snowflake (Azure Only).
- The new Multi Table Input component allows users to pull in data from multiple tables simultaneously to streamline their Transformation workflows.
Matillion ETL for Snowflake 1.37
Note: The Data Transfer component is now the preferred way to move files/objects between storage providers. Existing components such as S3 Get/Put, GCS Get will continue to work in existing jobs, but new jobs should use Data Transfer.
- New Data Loading orchestration components:
- All Data Loading orchestration components can now display data directly sampled from their source, allowing users to easily check their current configuration.
- All Data Loading orchestration components can now display the SQL generated by the component.
- Azure SQL Query component connects to a Microsoft Azure SQL Database. (Only for users with Matillion ETL instances hosted on Microsoft Azure)
- JDBC Table Metadata to Grid component connects to many types of JDBC database and can export the metadata from a source table into a Matillion ETL Grid Variable.
- A new Migration Tool can help move any number of Matillion ETL assets directly from one Matillion ETL instance to another.
Note: The preferred (and safest) way to upgrade is now to launch a new copy of Matillion ETL running the latest version, use the Migration Tool to move and validate the new version, before deleting the existing instance. In-place upgrade may be removed in future versions.
Matillion ETL for Snowflake 1.36
- A selection of new components to connect to various services:
- Shopify Query connects to the user’s Shopify account.
- Survey Monkey Query loads data from a SurveyMonkey database.
- Zoho CRM Query retrieves Zoho CRM data.
- Dynamics 365 Sales Query connects to the Sales service in Dynamics 365.
- Dynamics 365 Business Central Query connects to the Business Central services in Dynamics 365.
- DynamoDB Query loads data from Amazon’s DynamoDB.
- Blob Storage Unload Component (for users hosted on Azure) unloads data from your Snowflake warehouse into Azure Blob Storage.
- ORC and PARQUET file formats now supported in S3 Load.
- Many UX improvements including automatically connecting components on the canvas, improved variables workflow and new keyboard shortcuts.
- Selected Environments are now user-specific. Users can now specify their environments independently of one another.
- Users can now freely copy, cut, and paste jobs within a project.
- Autocompletion prompts now appear in many places when using Matillion ETL variables in code.
- CloudFormation Launch options (for users hosted on AWS)
- Data Lineage
- Allows you to understand the effect that your complex transformation jobs will have on your data. Track a column backwards to its source to determine where and how calculations are applied.
- This is an Enterprise Feature and thus is available to customers using large and xlarge instance types.
- New wizards to easily create Incremental Load Shared Jobs:
Matillion ETL for Snowflake 1.35
- New “Data Transfer” Component that boasts all the functionality of the existing S3 Get, S3 Put and Cloud Storage Put components, plus additional source and target destinations (Azure Blob Storage).
- In addition to AWS and GCP credentials, environments can now reference Azure credentials to interact with Azure services such as Blob Storage.
- A new “Apache Hive Query” component connects to your Apache Hive data warehouse.
- A new “LinkedIn Query” component connects to your company’s LinkedIn apps.
- A new “Bing Search Query” component connects to the Bing Search API.
- A new “Bing Ads Query” component connects to the Bing Ads service.
- A new “Dynamics 365 Sales Query” component connects to the Dynamics 365
- Allow uploading the native Microsoft SQL Server JDBC Driver (the bundled jTDS driver is often the fastest in scenarios where it works)
- New ‘Extract To New Job’ function available by right-clicking a selection of multiple components on the canvas. Allows users to instantly create new jobs from a group of components, tidying up workflows and helping to create reusable jobs.
- Support for running on Azure platform
- Launch Matillion ETL for Snowflake from the Azure Blob Storage Load” allows users to load data from Blob Storage into tables.
- Support for Snowflake’s AUTOINCREMENT option in the Create Table component. Particularly useful for easily creating a unique key column on a new table.
Matillion ETL for Snowflake 1.34.5
Matillion ETL for Snowflake 1.34
Important Notice: On upgrade, a background task will restructure the task history. During this time not all historic tasks will be available to view in the UI or API. The process only takes a few minutes in the general case but can take several hours if you have millions of run history items. This will require additional disk space (either on the instance or on RDS depending on your setup) so ensure you have at least 50% free space before attempting the upgrade.
- Shared Jobs:
- You can now turn your reusable orchestration jobs into their own components with their own parameters, help and Icon.
- Shared jobs can be packaged and distributed across multiple ETL instances with Import and Export.
- Historic Task Viewer:
- Previously completed tasks can be viewed on the canvas along with any parameter errors.
- You can understand the canvas state of a job and also see the jobs contained in a Shared Job.
- An “Unconditional” connector:
- Its now simpler to build orchestrations where the next orchestration step is run regardless of the success or failure of the prior step. This avoids use of extra “and” and “or” components to achieve the same thing.
- “Auto Debug” for all Data Loading components:
- Data Loaders come with the Auto Debug property. When switched on, allows users to choose between 5 levels of Debug Logging verbosity.
- Makes it easier to retrieve logging information without console access to the Matillion ETL Instance. Include these logs in your support requests for much faster turnaround!
- Warning: Can potentially consume large amounts of disk space. Do not leave this switched on unless directly in need of it!
- It is now possible to import, export and modify permissions via the API.
- You can now bulk export data to Amazon RDS sources with the RDS Bulk Output Component
- The Create Table component now supports Zuora Bulk Query:
- Loads data from Zuora into Snowflake using their Bulk API.
- OpenID Connect support for third party login providers:
- You can now configure Matillion ETL to authenticate with any Open ID Connect provider.
- Default support for Google, Microsoft and Okta plus a “Generic” option.
- The SQL Script component now supports multiple queries.
Matillion ETL for Snowflake 1.33.10
- Hot fixes for Salesforce OOM
Matillion ETL for Snowflake 1.33
Important: Queries using the Advanced Mode of the Google BigQuery Query Component will pass the SQL directly to BigQuery without any interpretation. If this causes any problem, please set the Connection Option 'Query Passthrough' to FALSE
Important: Users with the API role bypass some permissions checks (reads) during API calls.
- Open Exchange Rates Query component connects to the Grid Iterator allow iterating the values of a Grid Variable, similarly to iterating through a table of values.
- SQL Editor (in all Query components) now shows available Tables/Columns and Variables to help you author and test SQL queries from source systems.
- A new “Notices” V1 API endpoint allows you to query the current system notifications and post new messages which notify all users.
- A new “User Configuration” V1 API endpoint allows you to do user management via the Matillion API.
- Matillion no longer requires “listAllBuckets” permission (although this is still recommended).
- Job Variables (scalar and grid) now have a “Visibility” that determines how they are used elsewhere.
- All variables now have a description.
- 100+ bug fixes across all areas of Matillion ETL.
- A new Construct Variant component to create Snowflake variants from data columns.
- Transpose Rows component to aggregate multiple rows of data into a single output row.
Data staging (from all Query components) may use INTERNAL Staging so you don’t need to care where intermediate files are stored.
- Sequence Support has been added from ‘Environments’ context menu within the client. Users can create their own Snowflake sequences that are accessible by components for easy generation of unique numbers across their datasets (for example, as a way of introducing primary keys).
Matillion ETL for Snowflake 1.32.8
- Change: Allow Tomcat to start even if there is a corrupt schedule attached to a job. A bug fix to prevent the corrupt schedules attached to jobs in Matillion ETL 1.32 is currently in progress.
- Bug Fix: Prevented inappropriate authentication errors when using basic OAuth in the Zuora Query component.
Matillion ETL for Snowflake 1.32
IMPORTANT : Ensure you have a backup before you upgrade. Security configuration changes are applied on upgrade. These changes cannot be reversed, so do not use “yum downgrade” (or similar) to attempt to get back to versions prior to 1.32.
Enterprise Only: This version of Matillion introduces a new Permissions system that allows users to:
- Setup users with fine grained permission sets that can limit the 100+ core functions of the tool
- Provides default permission groups:
- Reader - Read only user who can’t modify a project
- Reader with Comments - Reader with ability to add notes to jobs
- Runner - A user who can execute but not modify jobs
- Scheduler - A user who can execute, schedule and change related config
- Writer - A user who can create ETL jobs but not delete projects
- Additional permission groups can be added at any time and are organised hierarchically making them easy to set up.
- A new suite of Grid Variable components are now included to make populating and manipulating them simpler - often without requiring any scripting:
- A new “SendGrid Query” component connects to the ElasticSearch Query” component to connect to the Magento Query” component to connect to the Zuora Query” component to connect to the GMail Query” component to connect to Google’s email service
- A new “Run Now” action has been added when defining a schedule
- Double-clicking a component on the canvas now opens the components “default” editor, if it has one. For example, double-clicking a Bash Script component will begin editing the script.
- Internal User: When using the “internal” security option tomcat user passwords are now hashed when stored on disk.
- External (Domain-based) Login: You can now encrypt your Realm Password with the AWS Key Management Service (KMS)
Matillion ETL for Snowflake 1.31.8
The "Google AdWords Query" component has been updated to support the latest Google AdWords API's.
Matillion ETL for Snowflake 1.31.7
Important (possible breaking change): API Profiles ("RSD’s") that handle paging may need to be tweaked to disable “auto” paging. Please see here for more details.
Important (possible breaking change): API profile limits are now applied. Where the default of 100 is set it will now be applied. This could affect API Query Components which previously ignored that limit.
- Zendesk Query orchestration component for loading data from the Mixpanel Query orchestration component for loading data from Xero Query orchestration component for loading data from the Dynamics 365 Query orchestration component for loading data from API Profile RSD Generator
- Accelerate the development of API Profiles using a new tool that automatically generates a basic XML “RSD descriptor” for any API endpoint, based on a sample of data returned.
- REST API Version 1 - Matillion ETL now has full API coverage:-
- You can now read/write more assets (JDBC Drivers, credentials, SQS configuration) as well as allowing finer-control of which resources to include.
- A map of the v1 API is available here.
- The “v0” api is still available and unchanged.
- Grid Variables System
- In addition to “scalar” (single-valued) variables, you can now define grid variables to hold lists and grids of values; use them wherever a compatible list or grid of values is required.
- Grid variables can be manipulated/modified in Python.
- You can pass values for grid variables when starting a job via SQS and/or the V1 API.
- You can now disable parts of an Orchestration job.
- Improved Matching in column mappings - Many transformation component “Column Mapping” parameters can now be automatically mapped, even when the input and output column names are similar but not identical.
- Viewless Architecture
- You may remove old views that were generated by previous versions of Matillion ETL for Snowflake, via the right-click menu of each environment.
- You may optionally define a “Default Role” in the Environment. Whenever a connection is required, the given username is first authenticated and then switched to the given role.
Matillion ETL for Snowflake 1.30.6
- The Google BigQuery Query component now supports standard SQL.
- New Intersect and Except transformation components
- New S3 Load Generator can inspect S3 files and generate Create Table / S3 Load components.
- Redesigned ""Task Info" panel and "Task" panel make it much easier to understand complex tasks both at run time and after job execution.
- Matillion variables can be defined and scoped at job level making jobs much more reusable. Variables can now be passed to and returned from jobs.
- New Quickbooks Online Query component to connect to the popular online accounting system.
- New Square Query component to connect to the payment system.
- New Google Custom Search component allows google search data to be ingested.
- All data-staging components can append rows to an existing table as well as creating new tables.