This proposal outlines a phased approach to broaden the scope of our Data Engineering capability, incorporating Microsoft Fabric as a unified analytics platform and evolving our approach to include foundational elements of AI Engineering. The goal is to create an agile, scalable, and intelligent data platform that supports operational analytics, predictive insights, and innovation. 1. … Read more
A novel about IT, DevOps and Helping your Business Win
🎯 Purpose This framework ensures that values in complex metadata structures (e.g. project earned value, invoice accounting, or employee attributes) are verified for presence even when using incremental loads. It protects against stale data persistence or silent value loss, using metadata-driven auditing. 🧩 Problem Context In many CDM views, values are upserted using procedures like … Read more
🎯 Objective To proactively expire or flag metadata rows that are no longer present in the source data even if their values haven’t changed, avoiding persistence of stale or misleading data. 1. 🏗 Recommended Table Extension Add these optional columns to any table using patch_upsert or put_update: 2. 📥 Staging Input Table – stage.project_values_current Your … Read more
🎯 Core Issue: 🔍 Risks of the Current Approach Scenario Problem Value disappears entirely You keep the last-seen value forever unless a new version overwrites Value moves to another key The old one is retained, and the new one appears as duplicate context Periodic reporting gaps Reporting shows a “stale” value that’s no longer present … Read more
SFIA Reference: METL Level 4/5 Data Engineer (Grade 3) Competency Description Drives iterative delivery and feedback-driven improvements in data engineering workflows. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources
SFIA Reference: TECH Level 4 Data Engineer (Grade 3) Competency Description Partners across teams to provide datasets optimised for analysis, reporting, or machine learning. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources
SFIA Reference: SCAD Level 4/5 Data Engineer (Grade 3) Competency Description Implements access controls, data classification, and auditing to ensure data security and regulatory compliance. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources
SFIA Reference: DESN Level 4/5 Data Engineer (Grade 3) Competency Description Improves query, job, and resource performance while managing cost within cloud data platforms. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources
SFIA Reference: METL Level 4/5 Data Engineer (Grade 3) Competency Description Implements automated, repeatable DevOps practices and infrastructure as code in data engineering. Learning Outcomes Evidence Requirements Level 4: Level 5: Suggested Learning Activities Recommended Resources