Roles & Permissions (Azure)
For Matillion ETL to detect Azure Blob Storage containers, additional credentials may be required. Matillion ETL can either use Instance Credentials or User Defined Credentials, the latter of which will require you to gather credentials from your Azure account and enter them into Matillion ETL.
Using Identities (Instance Credentials)
To use Instance Credentials, your Matillion ETL Virtual Machine (VM) must already be set up. If you wish to use a User Identity (as opposed to a System Assigned Identity, which is unique to the VM) then you will need to search for the Managed Identities blade on the Azure Portal and set one up.
If you have not already done so, please follow the steps below:
1. From the Azure Portal, click Virtual Machines and select the virtual machine containing your instance.
2. On the Virtual machines page, click Identity from the left-hand vertical menu, and then click User assigned and then click Add.
- If you wish to use a System Assigned Identity, click the System Assigned tab and set the Status to On. Make note of the Object ID.
- If you wish to use a User Assigned Identity, click User assigned and then Add a User Identity of your choice.
3. Next, on the Add user assigned managed identity dialog, select one or more user assigned managed identities you want to assign to the resource. Then, click Add.
4. Next, browse to Storage accounts on the Azure Portal and select the account(s) that contains Blob Storage that you wish for Matillion ETL to have access to.
5. Click Access control (IAM) from the left-hand vertical menu and then click Add, followed by Add role assignment.
6. In the Add role assignment dialog, set the Role as Storage Account contributor, and then select the desired user. Finally, click Save.
- For System Assigned Identities, set the Assign access to dropdown to Virtual Machine and select/search for the VM that you turned on System Assigned Identity..
- For User Assigned Identities, set the Assign access to dropdown to Azure AD user, group, or application and select/search for the User Identity you assigned to your VM.
The image below is the example while creating a new environment in the Matillion ETL instance.
Using Apps (User Defined Credentials)
Creating an App and Owning Storage Accounts To add Storage Accounts to Matillion ETL,we must first create an App. This requires a user with the 'Application administrator' directory role.
Navigate to the Azure Portal. The Microsoft Azure login screen will appear immediately. Enter valid login credentials to continue. The browser will then redirect to the Microsoft Azure dashboard. Click App registrations on the Azure services menu at the top of the screen.
If App registrations is not available on the Azure services menu, simply click More services, on the right of the menu, for a longer list of options.
On the App registrations page, click + New registrations on the menu at the top of the screen.
Now, in the Register an application window, provide details for the following fields:
- Name – provide a name for the app
- Supported account types – tick the checkbox next to Accounts in any organizational directory (Any Azure AD directory – Multi-tenant) and personal Microsoft accounts
- Redirect URI (optional) – paste the Callback URL (copied from the Manage OAuth window in Matillion ETL), then click Register
The browser will then redirect to the Overview window on the app's newly created dashboard. From here, copy the codes to the right of Application (client) ID and Directory (tenant) ID as they will be required later.
- Return to the storage account and select the blob account from the list of storage accounts and click Overview and then, click Containers.
- To add a new container, click the + Container button, give it a name and click OK.
If you haven't already, add a Blob Storage resource to this storage account.
Now, when you import details from your App into Matillion ETL, your client will be able to discover those buckets the App has ownership of. To use this App in your Matillion ETL client, see the next section.
Gathering Azure credentials
For a Matillion ETL instance to take advantage of Azure resources, you are required to provide credentials in the
form of a Tenant ID, which is unique to your Azure account, then a Client ID and Secret Key which are taken from a
Tenant IDFrom the Azure Portal, browse to Azure Active Directory → Properties and take the Tenant ID .
Client IDBrowse from the Azure Portal to Azure Active Directory → App Registrations, and select an App that is associated with your desired Storage Accounts. Take the Application ID as your Client ID.
- Browse from the Azure Portal to Azure Active Directory → App Registrations. Select the App associated with your desired Storage Accounts
- Next, click Certificates & secrets on the sidebar on the left. Then, in the Certificates & secrets window, click + New client secret.
The Add a client secret pop-up window will then appear. Provide details for the following fields:
- Description – provide a description of the client secret.
- Expires – tick the checkbox next to when the client secret should expire, then click Add.
Returning to the Certificates & secrets window, the new client secret will appear on the list in the Client secrets section. Copy the Value of the relevant client secret, as it will be required in Authorising for use in Matillion ETL.
- Make sure to copy the client secret value right away as it may appear only once.
- Additionally, when copying the client secret value, some browsers may add a space to the end of the string. Watch out for this as it will cause the credentials to fail.
You can test credentials using Project → Manage Credentials.
If instance credentials are available, you can Test them by clicking the Test button at the top of the dialog. This will check access to any services that Matillion ETL uses. You may continue even if the tests fail, however some parts of the product may be impaired or non-functional without appropriate credentials.
User Defined CredentialsAzure are listed, by name, under their respective tabs. New User Defined Credentials can be added by using the button, edited using the Icon or Deleted using the Icon by each entry. When Creating or Editing credentials, a Test button is made available in the new dialog to check the details before finalising your credentials.
For detailed information on Manage Credentials, please visit here.