This proposal outlines a phased approach to broaden the scope of our Data Engineering capability, incorporating Microsoft Fabric as a unified analytics platform and evolving our approach to include foundational elements of AI Engineering. The goal is to create an agile, scalable, and intelligent data platform that supports operational analytics, predictive insights, and innovation. 1. … Read more
Commercially sensitive data includes any information that, if compromised, lost, or disclosed without authorisation, could cause moderate financial, competitive, reputational, or operational harm to the organisation. Handling such data requires additional care to ensure compliance with internal policies, contractual obligations, and regulatory requirements. 1. Key Characteristics of Commercially Sensitive Data Commercially sensitive data typically falls … Read more
In its most basic definition, sensitive data is a specific set of “special categories”: Article 9 of the GDPR prohibits special category data from being processed unless you can rely on an exemption.
This draft framework provides a robust SLA for DataMart availability, balancing business needs, technical feasibility, and stakeholder communication. 1. Availability Target Set a clear uptime percentage that aligns with the criticality of the DataMart to your business processes. 2. Operational Hours Define the expected operational hours of the DataMart. Specify if it needs to be … Read more
Microsoft Fabric is designed to integrate multiple analytics workloads, including data engineering, data science, and business intelligence, within a unified platform. While our current Medallion structure already has a logical progression of data refinement (Bronze → Silver → Gold), there are key differences in architecture, tooling, and approach that we might consider while emulating a … Read more
1. Base DataViews & DataMart Architecture 2. CRM and Transactional Data 3. Data Security and Governance 4. Signposting of Insights 5. Metadata and Data Lineage 6. Master Data Management (MDM) 7. ETL/ELT and Data Integration Processes 8. Analytics and Data Modelling 9. Data Lifecycle Management & Archiving 10. Data Quality and Stewardship 11. Real-Time Data … Read more
Objective: Design, develop, deploy, and maintain Datamarts with a focus on performance, security, and usability. Steps: Data Discovery and Requirements Gathering Datamart Design and Security Planning MVP Development and Iteration Deployment and Go-Live Maintenance and Performance Testing
Objective: Efficiently acquire data from various sources and ensure secure, accurate, and timely transport to the data lakehouse and data warehouse. Steps: Data Source Identification and Mapping Data Profiling and Classification Data Acquisition and Integration Transport to Data Warehouse Testing and Validation
It is essential for BMT to take a holistic approach and adopt a combination of different strategies, such as Data Ops, Data Observability, Data Mesh, and Data Quality Checks to improve the quality of data. Despite the differences in focus, one common thing among the frameworks strategies above is the need for a unified governance … Read more
Standard Operating Procedures (SOPs) guide the DOM’s processes, ensuring efficiency and consistency at each stage. Purpose & Scope of the Data Operating Model Standard Operating Procedures Across Key Stages Data Planning SOPs Data Acquisition SOPs Data Processing & Maintenance SOPs Data Publish & Share SOPs Snapshot & Archive SOPs Data Flow and Governance within the … Read more