Joe Plumb - Let's Do

Custom event-driven ETL with SQL, Azure Functions, Data Factory and Event Grid

2022-08-20T00:00:00Z

Overview #

Metadata driven frameworks are very common in ETL scenarios - they make it faster to develop processes and provide more flexibility when processing data in your system. There are lots of ways to build and orchestrate these processes - Azure Data Factory has a metadata-driven copy wizard you can use to quickly build data movement pipelines, Microsoft MVP Paul Andrew has built an open source metadata framework on top of Data Factory called adf.procfwk, and many organisations build and maintain their own custom metadata-driven frameworks too.

I worked on a data warehouse migration project earlier this year which involved modernizing key portions of the workload from SSIS and SQL Server into Azure. This included extending the existing metadata-driven ETL framework to operate in the new cloud architecture. The project was a success and I learned a lot about migrating and modernizing complex data systems; unfortunately end-to-end job performance in the new cloud environment was degraded. We knew what the root cause of this issue was - a mix of insufficient bandwidth between on premises and cloud, and some shortcuts I made during the implementation phase to extract job metadata from the custom framework.

What were those shortcuts? Without getting too bogged down in the detail, the metadata parsing and job triggering was implemented in a data factory pipeline which, at it's highest frequency, could only be triggered once every minute. This was a stark difference to the original system which ran multiple times a second (!). I knew a more lightweight event-driven solution was possible, but ran out of time to implement it in the scope of the project. Naturally, this has been bugging me ever since - so I found some time over the last few weeks to re-visit and solve the problem.

Solution architecture #

Azure SQL bindings for Azure Functions was announced in preview at the beginning of this year and I always thought this would form part of the solution. Unfortunately you can't yet trigger a function on updates to a table - only input and output bindings are supported, but that's enough for a working solution:

Processing flow #

Metadata control table(s) with job status managed in SQL DB. SQL Server Agent Job manages job provisioning.
Azure Functions recurring Timer trigger ensures function polls the SQL table for target jobs event-driven processes.
Azure Function collects jobs from metadata table in “Started” state, updates state to “processing”, and publishes an event with job metadata to Event Grid topic.
Azure Data Factory pipeline with custom event trigger is triggered, parsing metadata from inbound event for use in pipeline.

The sample code for the pipeline, function, and database table are available on GitHub. For a more detailed runthrough of how to implement this in your own environment, keep reading below!

Prerequisites #

Local environment

Microsoft SQL instance - this can be any flavor of MSSQL. For ease of development in a Linux environment, I recommend running the SQL Server Linux container image with Docker, but any Microsoft SQL will do.
Azurite Storage Emulator - similarly, for local development of Azure Functions I recommend the Azurite Storage Emulator container, also available via container image.
Azure Functions - for more details on setting up your local environment for Azure Functions development check the documentation.
- Azure Functions Core Tools - The SQL bindings extension is part of a preview extension bundle (v4.x). See this GitHub issue for details on updating core tools in your local environment.
- Azure Functions Preview - Ensure that the requirements.txt of the deployed app specifies azure-functions==1.11.3b1.

Cloud environment

Event Grid Topic - Create an event grid topic in your Azure Subscription. Note the Topic URI and Topic Key.
Azure Data Factory - Create an Azure Data Factory

SQL #

After starting your database service, you're ready to create the few objects required for this demo. If you are using the Linux container image, there are some steps you need to go through to update the password after you've started the container, all detailed in the documentation.

Create Job Table #

Connect to your SQL database with SSMS or Azure Data Studio. Execute the below script to create a sample database and job table:

CREATE DATABASE test;

CREATE TABLE dbo.ToDo (
    [JobId] int not null,
    [JobStepId] int not null primary key,
    [JobName] nvarchar(200) not null,
    [JobStepName] nvarchar(200) not null,
    [JobStepType] int not null,
    [JobStepStatus] nvarchar(200) not null,
    [JobMetadata] nvarchar(200) null
);

Insert a sample job

INSERT INTO dbo.ToDo 
([JobId], [JobStepId],[JobName],[JobStepName],[JobStepType],[JobStepStatus],[JobMetadata]) 
VALUES 
(3012, 301201, 'Sample Job', 'First step of job', 6, 'Started', '{"jobmetadata":"examplepayload"}');

Take a note of your user ID and password. You'll need these when setting up the bindings in your Azure Function.

Azure Function #

Create new function #

Start by creating a new directory, then open this in VS Code. When you're in your empty directory, open the command palette (ctrl + shift + p), then find and select Azure Functions: Create Function. You'll then have to answer a series of prompts about the new function.

Select a language for your function project: Choose Python.
Select a Python alias to create a virtual environment: Choose the location of your Python interpreter. If the location isn't shown, type in the full path to your Python binary.
Select a template for your project's first function: Choose HTTP trigger. NB: in the initial design above, I recommend use of the timer trigger. For simplicity, we will use the HTTP trigger in this example function.
Provide a function name: Leave this as HTTPTrigger1.
Authorization level: Choose Anonymous, which enables anyone to call your function endpoint. You can find more information about access keys for functions here.

Update __init__.py with the following code:

import datetime, json, uuid
import azure.functions as func


def main(req: func.HttpRequest, jobItems: func.SqlRowList, outputEvent: func.Out[func.EventGridOutputEvent], jobItemsUpdate: func.Out[func.SqlRow] ) -> func.HttpResponse:
    rows = list(map(lambda r: json.loads(r.to_json()), jobItems))
    
    # Update status for inbound jobs
    for i in rows:
        i.update({"JobStepStatus": "ADF_Processing"})

    # Set ouputs
    outputEvent.set(func.EventGridOutputEvent(id=str(uuid.uuid4()), subject="adf-test-subject", event_type="adf-event-1",
                    data=rows, event_time=datetime.datetime.utcnow(), data_version="1.0"))
    jobItemsUpdate.set(func.SqlRowList(rows))

    return func.HttpResponse(
        json.dumps(rows),
        status_code=200,
        mimetype="application/json"
    )

Add input and output bindings #

Update the function.json file to include the 1 input (from SQL) and 2 output (SQL and Event Grid) bindings. You can learn more about bindings, including supported services in the Reference section of the documentation

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
          "get"
      ]
  },
  {
      "type": "http",
      "direction": "out",
      "name": "$return"
  },
  {
      "name": "jobItems",
      "type": "sql",
      "direction": "in",
      "commandText": "select [JobId], [JobStepId],[JobName],[JobStepName],[JobStepType],[JobStepStatus],[JobMetadata] from dbo.ToDo where [JobStepType] = 6 and [JobStepStatus] = 'Started' ",
      "commandType": "Text",
      "connectionStringSetting": "SqlConnectionString"
  },
  {
    "name": "jobItemsUpdate",
    "type": "sql",
    "direction": "out",
    "commandText": "dbo.ToDo",
    "connectionStringSetting": "SqlConnectionString"
  },
  {
    "type": "eventGrid",
    "name": "outputEvent",
    "topicEndpointUri": "MyEventGridTopicUriSetting",
    "topicKeySetting": "MyEventGridTopicKeySetting",
    "direction": "out"
  }
  ]
}

Note the commandText in the sql input binding - here we can see the query that will be executed on our database to collect incoming data. In this example, we're looking for job metadata when the JobStepType = 6 and JobStepStatus = 'Started'.

Add secrets to `local.settings.json` #

Notice in the function.json the connection settings you need to add - SqlConnectionString for the SQL database, MyEventGridTopicUriSetting and MyEventGridTopicKeySetting for the Event Grid. If using the Azurite Storage Emulator, you can add the known development connection strings

  "SqlConnectionString": "Server=localhost,1433;Database=test;User Id=$USERNAME;Password=$YOUR_SECURE_PASSWORD;",
  "MyEventGridTopicUriSetting": "https://$EVENTGRIDTOPIC.westeurope-1.eventgrid.azure.net/api/events",
  "MyEventGridTopicKeySetting": "$EVENTGRIDKEY"

NB: also add "PYTHON_ISOLATE_WORKER_DEPENDENCIES": "1" as per this GitHub issue

Azure Data Factory #

Create data factory pipeline #

Navigate to the Author tab of your data factory instance and create a new pipeline. Call it EventTriggeredPipeline.
Add a pipeline parameter called data of type string
Add two variables JobStepName and JobStepMetadata, both of type string
Add a ForEach activity onto the canvas. Navigate to the settings tab and in the Items box put @array(pipeline().parameters.data). What is this doing? This is setting up the details of what the ForEach loop will iterate over. In this example, it will be every row that satisifies the conditions of the SQL query that was added to the commandText in the sql input binding.
In the ForEach loop, add two Set Variable activities. For the JobStepName variable, set the value as @item().JobStepName, for JobMetadata, @item().JobMetadata. Your pipeline is now complete and should look like the below image:

Create Custom Event Trigger #

You are now going to create a custom event trigger to run the pipeline when events are written onto your Event Grid topic.

Click Add Trigger > New/Edit
At the top of the pane, select the dropdown box and click + New
Give the trigger a name e.g. EventTrigger
Change type to Custom Event
Use the Account selection method to select your subscription and Event Grid Topic name
Optional: Apply Subject filters to your trigger. E.g. Subject begins with adf
Click OK. This will bring up a new pane Trigger Run Parameters
For the data parameter, put @triggerBody().event.data as the value. This passes the Event data from the event on the Event Grid topic into the pipeline at execution time.
Click Save.
Once save is complete, click Publish.

Putting it all together #

You are now ready to run your first event-driven trigger of the data factory pipeline.

Start the Azure Function in your local environment (press F5 or click Run > Start debugging)
Trigger the Function via HTTP by navigating to the Function endpoint in your browser or calling via Postman e.g. http://localhost:7071/api/HttpTrigger1
Navigate to the Monitor tab > Trigger runs of your data factory and view the successful pipeline execution:

Conclusion #

In this post, I've covered how to get started triggering pipelines in Azure Data Factory using data from MSSQL, with custom event triggers, Event Grid, and Azure Functions. Let me know if this was useful, good luck and happy building!

- Joe

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

Processing manual data exports with Power Query and Dataverse

2022-06-24T00:00:00Z

I've recently been working on project with a requirement to ingest small datasets into Dataverse to serve a Power App. There were (unfortunately) no programmatic export options for this source data - exports were done manually via a reporting interface and gave the user spreadsheet extracts of 1000 records at a time. Additionally, data in the source system could be updated at any time, creating the opportunity for stale data and duplicate records in this downstream system. I needed to build a solution to process these extracts - merging all existing data extracts with the new extracts, and always promoting the latest data in the case of duplicate entries for a given record. This data then needed to be added to the existing Dataverse table, inserting new records and updating existing records (more here).

My initial instinct was to implement a solution using code with SQL and Python, but the solution was already entirely oriented around Power Platform (it was a Power App that was being developed!), so, I decided to take on the challenge and try and make use of the existing tools available: enter Power Query and Cloud Flows. (I initially tried to implement the whole solution with PowerQuery, however I couldn't figure out how to process the merge between the incoming data from the spreadsheets and the existing data in the target table - many thanks to my esteemed colleague Jonathan Ball for introducing me to Cloud flows and supporting me with this part of the solution!)

In this blog, I will walk through the steps to set up this process in your own environment to serve your own data processing use case. The walk-through will be broken into 4 parts; solution overview, ingesting files and data transformation, merging records from your staging table to the production tables, and creating the trigger flow.

1. Solution overview #

Let's take a look at the final solution design for what I delivered:

Processing flow #

Required data extracted from application manually via built-in data export tool
Exported file(s) are uploaded to SharePoint folder
File upload triggers PowerApps Data Flow(s) via Microsoft Flow
Data Flows process data to clean and conform data staging tables with Power Query. Records are de-duped, taking only the latest records from source for each unique record
Cloud flows used to compare all inbound records to existing records in Dataverse, inserting new records and updating records that already exist
Updated data available for consumption in PowerApps

2. Ingesting files and staging the data #

Creating and populating the landing zone #

First thing to do is provision the landing zone for inbound files. I used a SharePoint site in this project because these were already available in the environment and provided an easy, familiar interface for users to interact with.

Create one top-level folder e.g. etlfolder, with folders underneath this for each inbound dataset. The sub folders can be used to filter files in Power Query to ensure you are processing the right files with each flow (in situations where you are building more than one pipeline).
Upload some sample data files into this folder. For this example I will be working with these dataexamplex.xlsx spreadsheets which you can download from GitHub.

Staging the data with Power Query #

The next task is to create a new dataflow with Power Query to ingest and land the data from your files into the staging table:

Create the Dataflow #

In Power Apps, go to Dataverse > Dataflows and click New Dataflow +. Give the dataflow a name and click Create.
Select the data source - for this example, we will be reading the .xlsx files we have already uploaded to SharePoint, so select SharePoint folder. Enter your SharePoint site URL and configure the connection credentials. Click Next.
A preview of the files you have uploaded into the folder will appear. Filter the file list if necessary, click Combine, and wait for the Combine files dialog to appear. Select the sheet with your data in (e.g. Sheet 1), and click Transform Data. Power Query will create some helper queries to unpack the data from all the xlsx files you have uploaded to SharePoint. If you experience errors here, or need to adjust the logic at all because of different source file types etc, you can find more details on the Combine Files approach in the docs.

You should now see the Power Query window with some helper functions and queries created for you, and all the rows of data from your files. Next, we will process and clean the data to only get the latest rows from your files.

Cleaning data and filtering duplicates #

First, perform any additional data cleaning activities you require. Ensure data types are set correctly, remove rows with missing values, etc.
Order the data by your LastModifiedDate column in descending order by clicking the arrow next to the column header and clicking Sort descending.
Next, we will use the Table.Buffer function to force Power Query to load all the data into memory and ensure you always get the latest row when you filter out the distinct values - credit for this detail goes to Matt Allington of ExceleratorBI. Open the advanced editor for the query and wrap the last row with the Table.Buffer function e.g.
```
  #"Sorted rows" = Table.Sort(#"Changed column type", lastmodified)
```
becomes
```
  #"Sorted rows" = Table.Buffer(Table.Sort(#"Changed column type", lastmodified))
```
Finally, remove the duplicates on your ID column by right clicking the column header and clicking Remove duplicates
Click Next. In the map table dialogue, ensure the query you just created is set to load to a new table, and that the Unique primary name column is set to your ID column. Give the table a unique name. Check the Delete rows that no longer exist in the query output option. Click Next.
Leave the refresh as Refresh manually, and click Publish.

Once the dataflow has been published it will run for the first time automatically - verify that you see the records in your staging table.

3. Merge from Staging to Production table #

Create new Cloud flow by navigating to Flows > New flow > Automated cloud flow.
Give the flow a name, select When a dataflow refresh completes as the trigger and click Create.
Populate the trigger with the details of the Power Query flow you created in the previous step.
Build the rest of the flow with the following logic
1. List rows in the staging table with the List rows activity for Dataverse
2. Add a new step to list all rows from your production/target table with the same List rows activity for Dataverse. Then go to the advanced settings and in the Filter rows section, add a condition to compare the ID column name for the target table to the ID value from the first List Rows activity e.g. Type ContactId eq then click the Id value in the dynamic content window. Selecting the unique identifier from the Dynamic content section will embed this activity in an Apply to each loop.
3. Still inside the Apply to each loop, add a condition activity with one condition. This condition will establish whether the record from the staging table exists in the target table. The condition should use expressions. The left hand side should be empty(outputs('List_Contacts')?['body/value']), where List_Contacts is your second list activity. Select is equal to for the condition. The right hand side should be true (make sure you add this as an expression).
4. Finally, add the details to process the inbound records; in the If yes for the condition, add a Add new row activity and select the target table. Map the columns in the target table to the columns from the dynamic content pane. In the If no section, add an Update row activity, referencing the same target table, and adding the same mapping between the columns in the target table, and the columns in the dynamic content pane for the row from the first List rows activity. This will automatically add another Apply to each loop again - leave this as it is. NB: be sure that the ID column in your inbound data is mapped to the ID column on the target table.
Once you have completed the above actions, you should have a flow that looks like the below
Click Save.

4. Creating the trigger #

Finally, Because we want this process to execute every time we upload more data, we need to create a new Automated Cloud Flow that uses the SharePoint "When a file is created in a folder" trigger. This will trigger the dataflow to run, which will in turn trigger the Cloud flow to run to update the Dataverse table.

Navigate to flows in your environment and select `New Flow > Automated cloud flow":
Give your flow a name, and select the SharePoint "When a file is created in a folder" trigger. When the canvas loads, update the metadata in your flow to reference the new landing zone folder you have just created for the inbound data files. Then, add a Refresh a dataflow step below the trigger and add the references for the cloud flow you created in the previous step. You should now have a flow that looks like this:
The pipeline is complete! Save your flow and test the end to end process by uploading a new spreadsheet to the SharePoint site with some additional rows of data to check everything is configured correctly.

Conclusion #

In this post, I've covered how to build and automated pipeline to merge records from multiple point-in-time file extracts into Dataverse using Dataflows and Cloud flows in Microsoft Power Platform. Let me know if this was useful, good luck and happy building!

- Joe

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

Calling Microsoft Graph APIs via Logic Apps with service principals and app permissions

2021-12-06T00:00:00Z

I was doing a piece of work today where I needed to run some API calls against Microsoft Graph APIs to collect and process some information about service accounts (known as service principals in Azure AD parlance). Although there are lots of built in connectors for Logic Apps, there aren't any out of the box connectors for the Microsoft Graph APIs, so to build this integration we currently need to rely on the HTTP connector.

Thankfully, my colleague Tareq Saifi has written this article on the Microsoft Tech Community that explains how to call the Graph APIs using delegated permissions. This was enough to get me started however I hit some issues along the way that I wanted to document, as a note for future me and maybe useful to others too.

The main differences and issues I faced compared to Tareq's blog were:

needed to integrate via application permissions, as this was going to be a back end process to evaluate service principal registrations.
issues finding which permissions I needed to add to my service principal to perform the actions to solve my problem.
issues granting the permissions to the service principal to call the Graph APIs.
response from the oauth2/token API had a different schema.

I'll walk through each of these in turn through the remainder of this post.

Delegated permissions versus Application permissions #

The first thing that was different between what I was attempting to achieve and Tareq's blog is the subtle but important distinction when selecting the permission type when granting appropriate permissions to your service principal (step 1M in the original post) - you can select either delegated or application permissions:

Delegated permissions are used by apps that have a signed-in user present. For these apps, either the user or an administrator consents to the permissions that the app requests. The app is delegated with the permission to act as a signed-in user when it makes calls to the target resource.

Application permissions are used by apps that run without a signed-in user present, for example, apps that run as background services or daemons. Only an administrator can consent to application permissions.

Due to the nature of the service I was implementing, application permissions were the appropriate selection. This is then where the next issue came up - what permissions do I actually need to grant here?

Which permissions to grant? #

After I had selected application permissions, it wasn't immediately clear to me where find which actions would be authorized in the Graph API when when choosing which permissions to grant (step 1N from original post).

The answer lay in the Graph REST API specification. In this example you can see clearly which permissions allow you to call each API, in order of least to most privileged. The next issue was actually a blocker to progress this much further - I had insufficient permission to grant permissions!

Permission to grant permission #

The final hurdle was in granting the permissions to my service principal. Tareq covers this in his blog in step 1O - however this glosses over the fact that not every user has permission to change permissions in your Azure AD tenant. Checking the docs...:

The Azure account must have permission to manage applications in Azure Active Directory (Azure AD). Any of the following Azure AD roles include the required permissions:

Application administrator

Application developer

Cloud application administrator

Unsurprisingly I am not afforded any of these roles in the internal Microsoft AD tenant - so to build this demo I was at a dead end. I worked around this by using a different Azure AD tenant (where I host this site, in fact).

If I needed to work through this challenge in a tenant where this was a blocker, the solution would be to talk to the administrator of the tenant and agree to sign off on permissions for my App Registration to have the required permissions to perform the operations I needed. This block is here for a reason - a good permissions management strategy is all part of sound security posture.

A small schema change #

The final thing that appears to have changed since the initial post was that the schema for the response from the oauth2/token API was different - for me, it took the following structure:

{
    "properties": {
        "access_token": {
            "type": "string"
        },
        "expires_in": {
            "type": "integer"
        },
        "ext_expires_in": {
            "type": "integer"
        },
        "token_type": {
            "type": "string"
        }
    },
    "type": "object"
}

This was easy to manage with the Use sample payload to generate schema functionality in the Parse json activity.

Conclusion #

I've walked through some of the subtle differences in permission configuration to enable your service principals to interact with the Graph API. I've also highlighted some of the challenges I faced along the way to help accelerate others.

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

SQL Server Data Tools - A deeper look

2021-04-27T00:00:00Z

In my last blog, I introduced the importance of leveraging DevOps practices and principles for your infrastructure and analytics artefacts, as well as discussing two possible approaches for managing changes to a database as part of this process, and some tooling that you can use to bring this theory into practice.

In this blog, I will dive deeper into the database project itself – what it is, how to manage it, and give a little more context so you know what to expect as you get started yourself.

All screenshots have been captured from Visual Studio 2019 Community Edition, which is currently my preferred tool for working with SSDT, however the same tools are now available as part of Azure Data Studio!

Database projects #

Database projects are composed of the .sql files that make up the data model in your system, and several configuration files that SQL Server Data Tools (SSDT) uses to manage, control, and apply changes to your database. When starting a new project you have two options: you can create a new empty database project, or import a database schema from an existing database, .sql script file or a Data-tier application (.dacpac).

As the focus of this blog is on SSDT project files, I have created a new “SQL Server Database Project” by following the “New Project” wizard in Visual Studio.

After instantiating a blank project, your solution will look like this:

If you switch to the Folder view or look at this new directory in Windows Explorer, you’ll see these files:

Let’s dig into each of these files – what are they for and how should you manage them? Working from the bottom up:

`.sln` #

The .sln file maintains the state information for the project. It is text based (you can open it in a text editor of your choice and take a look), and contains data that the environment and IDE uses to find a load parameters and packages that are required to work with items in the project. It is designed to be shared with all developers who interact with the repository. For an in depth look at .sln files, check the documentation

✅ Store .sln in version control

`.sqlproj` and `.sqlproj.user` #

The proj files contain the properties for your project and is used by Visual Studio’s “Build” action to tell it what to build. The main difference is the .sqlproj contains properties that are specific to the project (e.g. which .sql files in the project should be included when creating the data model), and the .sqlproj.user file contains your specific user options for the project. As you work with your project to add and change database objects, you will see this file get updated. For more info, see this page in the documentation

✅ Store .sqlproj in version control
❌ Do not store .sqlproj.user in version control

`sqlprojAssemblyReference.cache` #

.cache files in a Visual Studio project are used to track the work that previous build processes have done to speed up subsequent builds in the local environment. Because these are specific to the local environment, the .cache files should not be checked in to version control. This is also reflected in the boilerplate .gitignore file.

❌ Do not store .cache in version control

Working with SSDT – development best practice #

Set the target platform #

Make sure this is set correctly –Visual Studio uses this to automatically detecting any error in your code and to build a compatible DACPAC for the target data store. Go into the project settings by double clicking on “Properties”, to make sure the correct target platform is set.

Organize your files #

This is not a requirement, but I find it easier to navigate and develop when I have the same folder structure I am familiar with from Management Studio. If you import a data model from an existing project or SQL source, you can choose what structure Visual Studio should use – I prefer the Schema/Object option.

When creating from scratch, you’ll have to build this into the project yourself. I aim for something like:

Schema name (e.g. dbo)
|- Tables
|- Views
|- Programmability
  | Stored procedures
|- Security
  | Users
  | Roles
  | Security Policies

I have seen others have separate folders for fact and dimension tables, too.

Use the designer to create or edit objects #

I’m a big fan of the built-in designer as it saves time searching through docs for valid syntax. To create a new object, right-click on the solution explorer and click Add > New Item. This will bring up a prompt where you can select the object types you wish to add, with some boilerplate code to get you started.

Conclusion #

I’ve talked through the files you can expect to run into when working with SQL Server Data Tools projects, made some recommendations on how to manage these files in your version control repository, and given some best practice for structuring the project and using Visual Studio features to speed up your development.

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

Collaborative Data Systems on Azure

2021-04-17T00:00:00Z

Use of DevOps practices and principles for solution design, implementation, and operations of infrastructure to support business has been increasing in popularity over the years. Agile development enables teams to move fast and focus on value; we are empowered to chart our own path to deliver functionality that delights users, and adapt to changing requirements over time. Everything-as-code is standard, and being able to rapidly provision additional capacity or develop new features to meet demand is taken as a given.

These techniques and expections from software development are now front and centre in the modern data ecosystem. From small scale analytics projects to enterprise data mesh architecture, these systems will almost always benefit from modern tooling and agile development practices. Effective analytics teams need reliable mechanisms to manage and promote changes across environments, collaborate with others, and ensure reliability of these changes before they get to production. Further, these mechanisms must account for the entire analytics lifecycle - the infrastructure deployment and management (i.e. compute, storage, networking), as well as the analytics artefacts themselves (i.e. data pipelines, dashboards, models) that are developed on top of the platform.

A data system without processes to manage the data artefacts within are likely to cost more to run, have reduced impact, and higher levels of technical debt.

In this post, I will look at what makes data systems different when it comes to agile development, review two patterns for version control with databases and discuss the benefits and considerations of both approaches, and highlight how other components of the data platform can also integrate with an overall delivery pipeline to manage data artefacts.

What makes DevOps with Data different? #

There is one subtle difference in a data system - that of state management. Application development does not typically have this problem as new versions or features simply supersede the old. You don't want to lose data when deploying changes, so ensuring this process is managed reliably is critical. In order to ensure the data remains in place and usable, the processes that promote changes to your data infrastucture need to take the existing state of the system into account. There are common two approches to managing this process: migration-based deployments and state-based deployments.

Migration-based deployment #

At a high level, migration-based deployments capture a “starting point” implementation of a database script, which represents the initial state. Changes and additions are then developed from this point as incremental migration scripts, which are checked into version control and applied to the database. The state of the database can be rebuilt by re-running the database initialization script, then all subsequent migration scripts in sequence on a new instance.

Migration-based deployment: changes are managed and deployed to the live database by applying a series of "migration scripts" on the system

State-based deployment #

A state-based deployment approach instead stores all database objects in source control as .sql files. New objects and updates to existing objects and are developed and added to these files in the repository, and tools are used to compare updated models to production databases to generate scripts to update the model in line with these changes. The scripts that are generated are similar to those that a developer would author in the migration-based deployment approach.

State-based deployment: Tooling is used to compare as-is and to-be state of the database - tooling like SQL Server Data Tools then generates a script to update the target database to the same state as the project model

Which is right for me? #

There is a case to be made for both approaches, and you need to consider the level of control you require over the changes you are making to your database.

I have a preference for state-based deployments as this approach relies on tools to generate the update scripts, leaving developers to focus on the changes that need to be made and hand off responsibility for the underlying state changes to the tooling. The state based deployment also reduces the barrier to entry as the repository itself is the control point for the state of the system, so updating objects in the database is similar to adding new functionality in a software project, which developers are likely to be more familiar with.

Migration based deployment does devolve full control over the changes that are made to developers, which may be preferable in some circumstances.

You might consider a hybrid approach too - state-based deployment tooling has the concept of pre and post deployment script steps, and these can be used to make manual changes to a deployment after the generated scripts have been created and executed in the database.

For further thoughts on the trade-offs between the two approaches, see this blog post from Samir Behara.

How can I start applying these processes to my Data System in Azure? #

If you opt for a state based approach to development, Azure SQL DB and Azure Synapse both have support for SQL Server Data Tools (SSDT). SSDT is free, and available as part of Visual Studio Community 2019. As part of the management of the analytics artefacts in your data system, you will also need to consider how you manage changes to data pipelines and dashboards. The support for this in Azure Data Factory is very comprehensive - the most recent update to this in February of this year has added support for the validation and publish of artefacts as part of your CI pipeline, meaning you can then manage the remaining deployment process in your pipelines. Automated deployment of artefacts in PowerBI takes a slightly different approach and use of version control is currently limited to use with OneDrive (i.e. not git) - you can read about deployment pipelines here.

NB: The "Database Projects" functionality in SSDT is also available in Azure Data Studio.

Conclusion #

We have looked at different approaches you can consider when adopting DevOps processes in your data system. I have focused on approaches to manage data artefacts in the database, and shared some further content and patterns to extend this process to manage lifecycle of the analytics artefact including the data pipelines and dashboard/reports.

Over the next few weeks I am going to write up some further information on this pattern, covering how you can get started with SQL Server Data Tools and Azure Synapse, and a deeper look at the SSDT Database Project. If you're looking for a hands on example, my Modern Data Warehouse with Terraform example I created earlier in the year to demonstrate how these systems can be managed and maintained should serve as a good starting point!

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

Fourth post - migration to 11ty.

2021-01-06T00:00:00Z

Well, there we are folks. Boris Johnson announces another national lockdown and it seems the best response I can come up with is ... refactoring my blog to 11ty!

It's been about 6 months since I updated my Angular-based site with blogging capabilities, which was a great learning experience, as well as an incredibly basic and fragile (very 2020) first attempt, lacking some notable features which I really wanted like tags, RSS, and section deeplinks. Enter, Eleventy (11ty).

In my initial search for frameworks I could use last year, I did review 11ty - in fact, the main feature I was after was the ability to write in markdown and have everything else handled for me, which 11ty does very well. I was instead attracted by the simplicity of a blog that I read, and the fact that I thought I would want some server-side compute element, and so stumbled down the rabbit hole of angular-ness, which was great fun but ultimately a classic case of over-engineering. At least I didn't deploy it on Kubernetes.

So, here we are. Visually similar, perhaps a little faster (thanks to this being pre-generated), and heaps more functionality, for a few hours work. The power of open source!

Building analytical data systems with Azure Synapse: A rough guide

2020-11-26T00:00:00Z

Over the last few months I've had the opportunity to spend more time working on enterprise analytics and business intelligence scenarios using Azure, based on the widely published and adopted Modern Data Warehouse architecture. It's been incredibly satisfying to refresh my visualisation and data modelling skills with PowerBI, and go deep on data orchestration and warehousing using Azure Synapse. As both a note to myself and a helping hand for others on the same journey, I wanted to reflect on what I've learned and share some key take aways after some hands on training and delivery of solutions for customers.

This blog will assume some familiarity with data processing patterns, tools, and techniques, and be broadly broken out into two sections; the process and approach to increase your likelihood of success, and technical considerations and decisions you'll encounter as you build out your MVP.

Process and approach #

Allocate time to learn the data #

If you're new into the problem space and datasets, make sure you allocate time to get up to speed with the data - depending on the complexity this could be anywhere from a few hours to a few days. If possible, include domain experts or people who are already familiar with the data structures and format on the team (or at least have a line of communication to them) to help answer questions that you will inevitably come up with as you get started.

BYOSQL (if you can!) #

One thing that is going to dramatically accelerate you is bringing your SQL from existing systems - table definitions (DDL), stored procedures, analytical queries.. anything you or existing users are already using is going to really help in the build out of the platform. It doesn't matter what system this is either, just make sure to allocate some time to make a few changes between the SQL code you have and T-SQL for Synapse as part of your sprint.

Reports and dashboards complete the vision #

There has been a lot of effort in recent years to make visualisation and reporting tools easy to use, and whatever tooling you have access to should make it relatively straight forward to build some impactful visualisations once the data model is in place. Being able to make good reports relies on a good understanding of the the data structure and clarity on the questions you or the user should be able to answer with the reports.

When thinking about what to build and how to build it, I generally recommend working with pen and paper - consider the questions you are looking to answer with the data, and start to think about what metrics and comparisons would be useful to see. Sketch out a low fidelity prototype (wireframe drawings are great for this) of the visualisation types, data, and overall report layout that you want to build to answer your questions. Once you've got something you're happy with, you can jump to the keyboard and start to build. This can be a great collaborative exercise if you're working as part of a team. You can also use these prototypes to walk your key stakeholders or sponsor users through what you're intending to build to get input and sign off (if required). This will radically reduce the time it takes to build your reports, and validate your thinking before you invest in implementing anything.

Don't leave without a map #

One of the key things to remember with any technical project is to plan ahead, know and agree the aims and outcomes, and review progress against this plan regularly as you go. Use the agreed scope and outcomes as your north star, as this is what you will be evaluating the outputs against when you come to the end of the sprint - you don't want to scupper the success of the team by getting sidetracked! Don't forget to also capture all the fun ideas you have along the way of things you could add - by capturing them you create an opportunity for these to be included in a future sprint.

Technical #

The flow of data through your MVP is likely to follow this common pattern. The focus of this post is on how to move through steps 1 to 3.

Architecture overview of data flow for Modern Data Warehouse pattern in Azure

At a high level, you're going to load a representative, raw sample of data from your source systems into Azure, use that to define and build your processing and transformation jobs and pipelines, build your target database structure in Synapse, then load your processed data into the warehouse. After that, you can work on connecting directly to the source systems for the full data load, and parameterizing your pipelines by implementing a tumbling window approach to incrementally load new data.

I've collated my notes on things you might want to consider and technical decisions you're going to have to make at each of these stages as you build out your use case.

Landing your data #

To get you going, manually land a representative sample from your source systems into your storage account in the quickest way you can. Azure Storage Explorer is a really useful tool to copy data to your cloud storage from your local machine. It's recommended to use folders to separate your data between layers and sources - for now, land the data as-is into a Raw folder, with a folder for each source. Maintain any other partitioning/folder/file structure that the data has - you'll need this to be the same structure as if you got it from your source system in order to make sure the processes that you develop will continue to work once you hook up to your live system.

/Raw/Source1/2020/11/01/file.extension

Example folder structure in the Raw layer of your lake

As you process the data, write it back out into the lake into different folders - you'll typically have at least a Cleansed and Curated layer in your lake which represents your datasets as you tidy them up, join them together, and apply more structure and value. Writing data back out into different layers means you can store all of your data at the lowest cost possible, re-process data from your Raw layer if necessary (e.g. to add in additional columns that were previously thought of as unnecessary), and provide analysts or data scientists access to your data for advanced analytics workloads, if appropriate. You can find more guidance on structuring your data lake for batch processing here. Data lake design is beyond the scope of this article, but this blog post written by my colleauge Nick Hurt goes into more detail of Data Lake organisation and design, too.

Data engineering and data structure #

There are different ways to do this depending on how you like to work and your programming skills - you can write SQL scripts and transform data using SQL Serverless or your SQL pool in Azure Synapse, take a no-code approach using Mapping Data Flows, or code your transformations in SQL, Scala, PySpark, or R, to run in Azure Databricks or Synapse Spark pools. Your choice will also depend on the complexity of the transformations you need to perform. My recommendation here is to go with what feels comfortable to you - there are comprehensive guides to getting started with each of the above approaches that I've linked to at the bottom of this article.

Things you'll be thinking about as you process your data are standardising on common formats, removing unnecessary columns, and applying cleaning rules like data standardization (e.g. set names to all capital letters) and data validation (e.g. validate product or customer IDs). This should be pretty easy if your data is already in a relational format and you're extracting in flat files (like csvs) - if your data is coming in in non-relational formats like json or xml, this may require you to unpack nested data which you can do using Mapping Data Flows or Spark.

One thing you may want to think about at this stage is how you intend to manage updates to data in your analytical store. If you want to persist history and changes to data over time, you can rely on the tried and tested method of Slowly Changing Dimensions.

Slowly changing what? #

Slowly changing dimensions! These are a set of patterns that enable you to track history and changes in your data. There are different types of slowly changing dimensions - you can implement logic to process these as you load new data into your warehouse, and templates are included to do this with Mapping Data Flows out of the box, too.

NB: This is not a requirement - this is an MVP after all! You can always come back and add this logic into your processing pipelines later on.

Distributing data in Synapse #

You'll also have to think about how the data is distributed in Synapse to optimize the operations your tables across the warehouse - depending on the data volumes and purpose of your table, there are different choices; round-robin and hash distributed tables, or replicated tables.

Round-robin distribution eveny and randomly distributes the data across all distributions. Loading data into round robin tables is fast, but query performance can be impacted as they can require more data movement. Consider round robin as a staring point, for tables that have no obvious column for hash distributing - it's also good for staging your data in Synapse before inserting to your production tables.
Hash distribution is used when it makes sense to group data in a distriubtion based on a certain value, minimizing data movement and improving query performance. You could distribiute on any column, perhaps country code, product line, or userid. You'll need relatively high cardinality for a column hash distribution (100s or 1000s of values), and the table size on disk is more than 2 GB.
For smaller, less frequently changing tables (e.g. dimensions) you might consider instead using replicated tables. This means a replica of the data will be stored in each distribution, saving shuffling of data which would degrade performance. Generally, 2GB or less is considered small, replicable table.

You can read more about distributed tables and design guidance in then documentation. This cheat sheet also gives a fantastic starter for designing your warehouse.

My final comment on this would be not to get too bogged down in the details here. Nothing that you build here will be fixed in stone, and you can expect to refine the decisions made as you address any performance issues at a later stage.

Loading from the lake into Synapse #

Once your data is prepared and in the right structure, and your target tables are created in your Synapse SQL pool, load your data from your storage account into Synapse using the copy activity. It's best practice to load your data into staging tables before inserting into production tables. Within the settings of your activity, you'll find an option of the function that should be used; PolyBase and the COPY command both use a parallel load routine, making them very efficient for big loads. Bulk insert is essentially batched INSERT statements of your data, so PolyBase or COPY are the preferred choices here.

Security and Authentication #

You can authorize users into Synapse with 2 authorization types; Azure Active Directory (AAD) Authentication, or SQL Authentication. There are a few steps to follow in order to enable Active Directory authentication, and these steps need to be performed by an administrator. Its best practice to use AAD authentication, and while it might feel a little more onerous to set up, the additional security it affords you is worth it.

Other performance tuning tricks to note #

Ordered clustered column store indexes can be used to further improve performance - patterns where ordered CCI will help are documented here.
Materialized views provide a low-friction mechanism for increasing performance by pre-calculating and maintaining the results of a view, rather than calculating the results at query time. This is all managed for you by Synapse, meaning the views get automatically refreshed as data gets loaded into the underlying tables. These can be a performant way to provide aggregations and calculations for downstream consumers.
Result set caching can also improve query performance and reduce compute resource usage. The result set cache has 1TB storage per database, and requires a user with the dbmanager role to turn it on. The cache will be invalidated if the underlying data changes too - the oldest results get removed from the cache to make space for new ones. NB: row level security cannot be applied with Result Set Cache on.

Conclusion #

The above should help you understand what you need know when you're considering building out a data analytics capability using Azure Synapse (or any comparable technologies). If you're considering trialling this for yourself, I've recently discovered this useful guide that walks through the PoC process in a lot more detail that might be helpful, too.

Please message me on Bluesky if this has been helpful, or useless, or if you have any questions or comments!

Serverless functions with Azure and Python

2020-08-25T00:00:00Z

I've been working with Azure Functions since late last year - my first real opportunity to work with them came about during the App Modernization with NoSQL Openhack, where some of the exercises required attendees (i.e. me!) to implement an event sourcing pattern in Azure. For so long before this point, I had been including them as a component in my architectures and systems design - how hard could they be?

Getting Started #

The main hurdle for me with getting started was understanding the concepts that make up the function, and the boilerplate folder structure that comes out of the box when you initialize your project. Thankfully, one of the things it feels like there has been a great focus on with Azure Functions is the developer experience. Once I had my first couple of functions up and running, I began to find them incredibly intuitive. Conceptually, there are only 2 things you need to understand - triggers and bindings:

Triggers define what needs to happen in order to make your functions run. These could be events, requests, or a schedule. Each function has one trigger.

Bindings are connections - I think of them as the services I want to "glue" my function to. You can have input and output bindings, and these can be the same or different types of services.

I find it helpful to focus on the data movement required to solve my problem - where does the data come from? Where does it need to go? What is the resulting action or output? From there, Inputs can be mapped into triggers (e.g. HTTP inputs) or bindings (e.g. fetch data from storage/a database), and similar for outputs (e.g. write the output to a datastore, return values over HTTP). Practical examples always help; let's take a look at building a function from scratch to understand these in more detail.

Prerequisites #

Before you get started, you'll need to make sure you've got a few tools installed. Firstly, you'll need a text editor - I recommend Visual Studio Code as I find it quick to work with and it's got a great ecosystem of extensions. You'll want to install a couple of these, namely Azure Functions and one for the programming language you are working with, in our case, Python. You'll of course need Python installed (3.8, 3.7, and 3.6 are supported by Azure Functions). The final thing you'll need is Azure Functions Core Tools, which includes a version of the same runtime that powers Azure Functions, which enables you to run Functions in your development environment.

With the above installed, you're all set!

Creating your first function #

Select a language for your function project: Choose Python.
Select a Python alias to create a virtual environment: Choose the location of your Python interpreter. If the location isn't shown, type in the full path to your Python binary.
Select a template for your project's first function: Choose HTTP trigger.
Provide a function name: Leave this as HTTPTrigger1.
Authorization level: Choose Anonymous, which enables anyone to call your function endpoint. You can find more information about access keys for functions here.

You'll now see a bunch of files created in your once empty directory. To run the function locally, start debug mode in VS Code (Run > Start Debugging, or press F5), or type func start in your terminal. This will start the local emulator for Azure Functions and enable you to test your code. See the output of the function by heading to the given url in your browser:

http://localhost:7071/api/HTTPTrigger1?name=World

I find this functionality to be really useful - it allows me to iterate on my code as I go and quickly validate the function is working as expected. But where does the code go? And how do you add or change the triggers and bindings for the function? Many files are generated when you create the templates, but there are only a handful of files you need focus on to begin with.

The three files you need to know about #

Demystifying the files that are automatically generated by the extension will help you get to outcomes faster. The three files you should focus on first are:

local.settings.json - this is used by Azure Functions Core Tools to connect to other services in Azure when you're running your functions project locally.
function.json defines the triggers and bindings for your functions. These are very thoroughly documented in the triggers and bindings reference, and you can often re-use code from here with minimal changes. Notice that $return is referenced in the output - this is the return value of the main function. Using this, you can pass the output of your function to the output binding.
__init__.py is where your code lives. If you've added additional bindings on top of the starter code make sure to , reference them in the main function, and you're away!

Deploying your first function #

When you're happy with your function, you're only a few steps away from deploying it to the cloud - the instructions below demonstrate how you can do this using the Azure Functions Extension in VS Code.

Choose the Azure icon in the Activity bar, then in the Azure: Functions area, choose the Deploy to function app... button.
As before, follow the prompts and provide the following information:
- Select subscription: If prompted, select the subscription to use.
- Select Function App in Azure: Choose + Create new Function App, or an existing one if you already have a service provisioned (NB: Deploying to Azure Functions with the extension will replace any existing functions with those that you push)
- Enter a globally unique name for the function app: Type a valid name into the URL path (the name you type will be validated to make sure that it's unique).
- Select a runtime: Choose the same version of Python you've been working on locally.
- Select a location: For better performance, choose a region near you.
The deployment will complete in the background - you can follow the progress by clicking the "Stream Logs" button on the prompt in VS Code. A notification will be displayed once the function app is created and deployed.
That's it! Your function is now deployed to Azure. You can see it in action by going to the function URL and passing parameters, just like we did with the local emulator, e.g.

http://$YOUR_FUNCTIONS_SERVICE_NAME.azurewebsites.net/api/HTTPTrigger1?name=World

There's a more comprehensive set of instructions (with more screenshots too) available in the documentation, if you'd like more details on any of the above steps.

Taking it further #

If you've got this far, you should be good to go! I've included some links and a summary of some other projects and repositories that might help you get started with your use case below.

Event sourcing with Cosmos DB and the Change Feed - As part of the Openhack that got me started, I pushed some starter functions for working with Cosmos DB to this repo. Check this out if you're looking for some examples of getting data from Event Hubs to Cosmos DB, or working with the Change Feed.
Creating Serverless APIs to share data from Azure Blob - After sharing this getting started content with my team, a colleague of mine asked whether Azure Functions would be a good fit to build a REST API for data sharing. Using a HTTP trigger and Azure Blob storage input binding, I was able to knock together this example.
End to end tutorial - This post covers the basics, and should cover all you need to know to get started. If you want to go into more detail, there is a brilliant end to end getting started tutorial in the documentation.

On my to do list #

One pattern I haven't yet had the opportunity to road test is the integration between Azure Machine Learning and Azure Functions. I can see this being a really powerful pattern for peaky, unpredictable workloads, or for any applications where deploying to Kubernetes would add unnecessary complexity.

Conclusion #

In this post, I've covered how to get started with Azure Functions and Python, including some references to other useful material and some GitHub repositories for examples. Let me know if this was useful, good luck and happy building!

- Joe

Has this helped you get up to speed with serverless? Have you deployed or do you want to know how to deploy ML models to the cloud using functions? Contact me on Twitter if so, or if you have any questions or comments!

✨Hey world!✨

2020-07-31T00:00:00Z

Note: this blog was originally hosted as an Angular app - we are now live with 11ty - woo!

Here we are then! First blog post going live in true mvp style. I am still adding to the structure, formatting and pipelines for the site, and am still missing some basic functionality and styling, so expect this post to evolve over the next week or so.

I thought quite a lot about how I wanted this to come together and what tech stack I wanted to use. I took a look at some static JAM-stack frameworks like 11ty and Gatsby, but eventually decided on Angular for a few of reasons - I wanted something that had a server-side component as I want to have the option of developing my own and integrating with other services, porting my existing Bootstrap site into Angular seemed relatively straight forward (it was! I was plesantly surprised for a Javascript noob), and, I found this useful blog that walked through how to do just what I was looking to achieve. Quite meta .. a blog about building a blog based on a blog post about building blogs .. but here we are.

My aim is simple. Here, I will document my learnings and thoughts about technology - predominantly analytics, big data, machine learning, and probably some coffee too - things I certainly spend a lot of time thinking about, and, in time, who knows, they may be of use to someone else!

So it's not perfect, and there are many like it, but this blog is truly ✨mine✨. And I'm looking forward to continuing to chart my path and document my journey!

- Joe

Joe Plumb - Let's Do

Custom event-driven ETL with SQL, Azure Functions, Data Factory and Event Grid

Overview #

Solution architecture #

Processing flow #

Prerequisites #

SQL #

Create Job Table #

Azure Function #

Create new function #

Add input and output bindings #

Add secrets to local.settings.json #

Azure Data Factory #

Create data factory pipeline #

Create Custom Event Trigger #

Putting it all together #

Conclusion #

Processing manual data exports with Power Query and Dataverse

1. Solution overview #

Processing flow #

2. Ingesting files and staging the data #

Creating and populating the landing zone #

Staging the data with Power Query #

Create the Dataflow #

Cleaning data and filtering duplicates #

3. Merge from Staging to Production table #

4. Creating the trigger #

Conclusion #

Calling Microsoft Graph APIs via Logic Apps with service principals and app permissions

Delegated permissions versus Application permissions #

Which permissions to grant? #

Permission to grant permission #

A small schema change #

Further reading #

Conclusion #

SQL Server Data Tools - A deeper look

Database projects #

.sln #

.sqlproj and .sqlproj.user #

sqlprojAssemblyReference.cache #

Working with SSDT – development best practice #

Set the target platform #

Organize your files #

Use the designer to create or edit objects #

Conclusion #

Collaborative Data Systems on Azure

What makes DevOps with Data different? #

Migration-based deployment #

State-based deployment #

Which is right for me? #

How can I start applying these processes to my Data System in Azure? #

Further reading on this topic #

Conclusion #

Fourth post - migration to 11ty.

Building analytical data systems with Azure Synapse: A rough guide

Process and approach #

Allocate time to learn the data #

BYOSQL (if you can!) #

Reports and dashboards complete the vision #

Don't leave without a map #

Technical #

Landing your data #

Data engineering and data structure #

Slowly changing what? #

Distributing data in Synapse #

Loading from the lake into Synapse #

Security and Authentication #

Other performance tuning tricks to note #

Further reading #

Conclusion #

Serverless functions with Azure and Python

Getting Started #

Prerequisites #

Creating your first function #

The three files you need to know about #

Deploying your first function #

Taking it further #

On my to do list #

Conclusion #

✨Hey world!✨

Add secrets to `local.settings.json` #

`.sln` #

`.sqlproj` and `.sqlproj.user` #

`sqlprojAssemblyReference.cache` #