This proposal outlines a phased approach to broaden the scope of our Data Engineering capability, incorporating Microsoft Fabric as a unified analytics platform and evolving our approach to include foundational elements of AI Engineering. The goal is to create an agile, scalable, and intelligent data platform that supports operational analytics, predictive insights, and innovation. 1. … Read more
🎯 Purpose This framework ensures that values in complex metadata structures (e.g. project earned value, invoice accounting, or employee attributes) are verified for presence even when using incremental loads. It protects against stale data persistence or silent value loss, using metadata-driven auditing. 🧩 Problem Context In many CDM views, values are upserted using procedures like … Read more
🎯 Objective To proactively expire or flag metadata rows that are no longer present in the source data even if their values haven’t changed, avoiding persistence of stale or misleading data. 1. 🏗 Recommended Table Extension Add these optional columns to any table using patch_upsert or put_update: 2. 📥 Staging Input Table – stage.project_values_current Your … Read more
🎯 Core Issue: 🔍 Risks of the Current Approach Scenario Problem Value disappears entirely You keep the last-seen value forever unless a new version overwrites Value moves to another key The old one is retained, and the new one appears as duplicate context Periodic reporting gaps Reporting shows a “stale” value that’s no longer present … Read more
Security Reminder Keep your SAS URL secure! Avoid sharing it with unauthorised individuals, as anyone with the link can perform the permitted actions. Renewing the SAS Token Install Azure Storage Explorer Connect to Azure Storage A SAS (Shared Access Signature) URL provides secure, delegated access to resources in an Azure Storage account. The SAS URL … Read more
1) What is API API stands for Application Programming Interface. That means it is an interface between two applications/computers that enables them to communicate with each other through some program. You can imagine the API as a postal pigeon delivering your request to the other side and receiving what you (hopefully) expect. In order for the … Read more
When evaluating an API for a platform, asking the right questions is critical for ensuring the platform can meet your needs. Here are some important questions to consider: 1. Authentication & Security 2. Data Access & Structure 3. Documentation & Support 4. API Rate & Limits 5. Performance & Reliability 6. Customisation & Extensibility 7. … Read more
When comparing the integration of CSV files and APIs with the DataWarehouse, several factors impact data integrity, including data accuracy, consistency, security, and the ability to handle large datasets. Below are some key comparisons and potential risks associated with each method: 1. Data Accuracy and Consistency CSV Integration: API Integration: 2. Data Security CSV Integration: … Read more
Create Parameters Create two parameters… Use a Token on myBMT The token allows the Datawarehouse to identify with the source of the report, as PowerBI cannot send environment details. (answers on a postcard please, if you know how) So the token is used to give the report a unique identifier myBMT | Token (bmt-dwh-uks-app-my.azurewebsites.net) Navigate … Read more
Set a goal to seamlessly integrate data from diverse sources and platforms within the organisation. This could involve building connectors, APIs, and data pipelines to facilitate smooth data flow between different systems and applications.