Connecting Azure Data Factory code to an external database table

In this article I will talk about how to connect Azure Data Factory (ADF) to a database table. This can be surprisingly complex, so I will start with the simplest version and work towards more complex versions. I won't go into connecting ADF to other types of data store such as APIs, blob storage etc, … Continue reading Connecting Azure Data Factory code to an external database table

Improving testability and observability of look-ups in data pipelines

Often in data pipelines (ETL or ELT pipelines for feeding a data warehouse, data science model etc.) we need to look up reference data that relates to the main flow of data through the pipeline. If this isn't done carefully, there can be problems for checking how the system is running. Before the system is … Continue reading Improving testability and observability of look-ups in data pipelines

Automating the deployment of SSIS packages

Introduction This post describes how I have automated the deployment of SSIS packages to different environments (test, staging, production etc.)  There are definitely other ways of doing it, and some of these are possibly better - I think that this is the oldest-school way possible.  I will describe our context which I hope will explain … Continue reading Automating the deployment of SSIS packages

From Excel to SSIS, Part 5: Joining streams of data

This article is part of a series: From Excel to SSIS Getting started File Save As (a kind of Hello, World! program for SSIS) Filtering rows Grouping and calculating Joining streams of data 0. Introduction This article will cover another very useful chunk of SSIS - joining streams of data together.  Often the value of two bits … Continue reading From Excel to SSIS, Part 5: Joining streams of data

From Excel to SSIS, Part 4: Grouping and calculating

This article is part of a series: From Excel to SSIS Getting started File Save As (a kind of Hello, World! program for SSIS) Filtering rows Grouping and calculating Joining streams of data 0. Introduction Over the articles in the series, we have been gradually working our way towards useful and interesting SSIS code.  So far we … Continue reading From Excel to SSIS, Part 4: Grouping and calculating