In a previous post, we laid down the foundational principles of a Data Mesh, and touched on some of the problems we have with the current analytical architectures. In this post, I will go deeper into the underlying principles of Data Mesh, particularly why we need an architecture paradigm like Data Mesh. Let’s start with why we need a paradigm like Data Mesh. Why do we need Data Mesh? In my previous post, I made the bold claim that analytical architectures hadn’t fundamentally progressed since the inception of the Data Warehouse in the 1990s. That’s three decades of the same thinking: ingest data into a central data repository, clean and transform data, serve data for consumption. ...

To be able to properly describe what Data Mesh is, we need to contextualise in which analytical generation we currently are, mostly so that we can describe what it is not. Analytical Generations The first generation of analytics is the humble Data Warehouse and has existed since the 1990s and, while being mature and well known, is not always implemented correctly and, even the purest of implementation, comes under the strain of creaking and complex ETLs as it has struggled to scale with the increased volume of data and demand from consumers. This has led to systems being maintained by a select few in an organisation who are able to understand the system, in its entirety or even partially. Where once it provided value, the benefits posed to many mature organisations is, perhaps, diminishing. ...

At SQL Bits, Europe’s largest data conference, I presented a Tabular Automation with TMSL and PowerShell. By now, I was a seasoned speaker but was still quite nervous, especially as the attendees of my session included members from Microsoft’s development team. Feedback from them, was that it was nice to demonstrate how TMSL could be used outside what it was originally intended to do. Video to the session can be found on the SQL Bits website. ...

At SQL Bits, Europe’s largest data conference, I presented a Deep Dive Into Data Lakes. It was my first solo speaking session and was the 08:00 session on the Saturday - right after the Friday night party. Video to the session can be found on the SQL Bits website.

Data Lakes are the new hot topic in the big data and BI communities. Data Lakes have been around for a few years now, but have only gained popular notice within the last year. In this blog I will take you through the concept of a Data Lake, so that you can begin your own voyage on the lakes. What is a Data Lake? Before we can answer this question, it’s worth reflecting on a concept which most of us know and love – Data Warehouses. A Data Warehouse is a form of data architecture. The core principal of a Data Warehouse isn’t the database, it’s the data architecture which the database and tools implement. Conceptually, the condensed and isolated features of a Data Warehouse are around: ...

What comes to your mind when you hear the words forecasting, forecasts etc? Invariably, you’ll think of weather forecasts. But forecasts are much more than that. Forecasting is the process of making predictions of the future based on past and present data and analysis of trends. It’s a process that has existed for millennia, though often with dubious methodologies… Instead of looking in to a crystal ball to predict the future we are going to employ the power of statistics! ...

Some systems use a single currency as a base, which is something that I noticed recently when working with IBM Cognos Controller, e.g. USD to convert local currencies into. But what if you want / need to rebase into another currency but still retain the original base? This doesn’t appear to be easy to achieve within Cognos Controller itself, but it is achievable within SQL and a wider ETL framework. ...