This proposal outlines a phased approach to broaden the scope of our Data Engineering capability, incorporating Microsoft Fabric as a unified analytics platform and evolving our approach to include foundational elements of AI Engineering. The goal is to create an agile, scalable, and intelligent data platform that supports operational analytics, predictive insights, and innovation.
1. Current State
- Data Engineering: Strong foundation in medallion architecture, metadata-driven modelling, pipeline orchestration, and warehouse optimisation.
- Fabric Adoption: Ongoing adoption of Microsoft Fabric for OnDemand SQL, capacity-based cost management, and integration into existing Gold/Platinum layers.
- AI Use Today: Light, informal use of AI (e.g., Copilot, ChatGPT) as augmentation tools. No dedicated AI engineering pipelines or responsible ML practices in place.
2. Strategic Drivers
- Demand for faster insight and automated decision support.
- Increased focus on predictive and generative AI capabilities across the business.
- Opportunity to consolidate tooling under Microsoft Fabric (Data Factory, Lakehouse, Notebooks, ML).
- Desire to raise the profile of Data Engineering and position it as an enabler of responsible AI.
3. Vision
“From Data Pipelines to Intelligence Pipelines: Making Data Engineering the engine that powers decision intelligence at scale.”
4. Roadmap: Capability Expansion
Phase 1: Foundation – Fabric First
- Finalise Fabric architecture (we’re nearly there).
- Establish robust governance (cost controls, workspace rules, RBAC).
- Promote use of
Semantic Modelsfor analytics delivery.
Phase 2: Enablement – AI Readiness
- Define what AI Engineering means for BMT: likely a blend of MLOps, applied analytics, and AI augmentation.
- Train the team in:
- Responsible AI (e.g., Microsoft Responsible AI Standard).
- ML pipelines (e.g., notebooks, PyTorch/Scikit-Learn in Fabric).
- AutoML and low-code AI in Fabric.
- Create “Hello AI” use cases: e.g. anomaly detection, trend forecasting, or classification.
Phase 3: Integration – ML into Production
- Build a small ML Ops framework using Fabric Pipelines & Notebooks.
- Establish model governance practices.
- Partner with domains (e.g. Projects, Future Business, Customers) for AI-driven enhancements.
5. Proposed Team Strategy
- Ali continues as Lead Fabric Engineer.
- Add an AI Engineering learning stream into the Data Wrangler Academy.
- Define roles that bridge between Engineering and AI (e.g., “Analytics Engineer”, “Applied Data Scientist”).
- Consider secondments, short projects, or bringing in an AI coach.
6. Risks & Considerations
- AI is not a magic bullet; define clear success metrics.
- Avoid “AI theatre” – focus on useful, trusted outcomes.
- Balance between cost (especially Fabric capacity) and innovation appetite.
- Need for ongoing learning and awareness of ethics and bias in data/AI.
7. Recommendations
- Approve a 3–6 month Fabric & AI Engineering enablement programme (Fabric is now, but AI is after IFS Cloud FY26 Q1).
- Align with business sponsors for two pilot AI projects. (i.e have another go with 18 months of data ;))
- Present roadmap at next Show & Tell to engage wider interest.
- Use this momentum to formalise a Data & AI Strategy under our data governance framework.