Julian.Kellett This white paper proposes a comprehensive Data Protection Impact Assessment (DPIA) framework for the Data Warehouse, detailing the potential commercial and personal data risks across individual data marts—including Projects, Employee, and Customer—and introducing a “Defence by Design” methodology to mitigate these risks effectively. Objectives Data Marts Overview and Risk Assessment 1. Projects Data Mart 2. … Read more
Additions customer_orderline_202501 This release introduces a new version of customer.orderline, evolving from the current beta version. The latest update incorporates a fully JSON-managed field structure, enhancing flexibility, data integrity, and future scalability. Key Changes & Enhancements Impact on Data Consumers Alterations project_details_plus_CAD This update continues the ongoing improvements to mapping accuracy and data consistency within … Read more
Commercially sensitive data includes any information that, if compromised, lost, or disclosed without authorisation, could cause moderate financial, competitive, reputational, or operational harm to the organisation. Handling such data requires additional care to ensure compliance with internal policies, contractual obligations, and regulatory requirements. 1. Key Characteristics of Commercially Sensitive Data Commercially sensitive data typically falls … Read more
In its most basic definition, sensitive data is a specific set of “special categories”: Article 9 of the GDPR prohibits special category data from being processed unless you can rely on an exemption.
Core Views and their Standard Structures Each DataMart will be built on six primary views: 1. Details (or Core) View Purpose: Schema: Column Name Data Type Description company VARCHAR Primary entity identifier object_id VARCHAR Unique object identifier (project_id) seq_json JSON Structured metadata reference for linking name VARCHAR Name of the project/entity description TEXT Brief descriptive … Read more
Standardisation of DataMart View Queries for SQL OnDemand To enhance query consistency and readiness for SQL OnDemand, we have implemented a major standardisation across all DataMart view queries. Key Enhancements: ✅ Uniform Aliases for Data Sources – All table references now include consistent and meaningful aliases, improving readability and maintainability. ✅ Standard Positioning of — … Read more
This draft framework provides a robust SLA for DataMart availability, balancing business needs, technical feasibility, and stakeholder communication. 1. Availability Target Set a clear uptime percentage that aligns with the criticality of the DataMart to your business processes. 2. Operational Hours Define the expected operational hours of the DataMart. Specify if it needs to be … Read more
Microsoft Fabric is designed to integrate multiple analytics workloads, including data engineering, data science, and business intelligence, within a unified platform. While our current Medallion structure already has a logical progression of data refinement (Bronze → Silver → Gold), there are key differences in architecture, tooling, and approach that we might consider while emulating a … Read more
Purpose The Customer Invoice Item data mart is designed to provide detailed insights into invoice-level transactional data for customers. It organises, validates, and enriches raw data from the source system into a structured and accessible format, enabling comprehensive analysis and reporting. Use Case Owner: Soon Tan Typical Use Case: Style: Fact, Each row represents a unique Invoice … Read more
An interim release with no functional changes Notes 1. 🔴 MISSING IN PRD CONTAINER: customer_details_CAD.sql2. ⚠️ DIFFERENT: customer_details.sql3. ⚠️ DIFFERENT: project_cost_CAD.sql4. ⚠️ DIFFERENT: project_details_plus_CAD.sql5. ⚠️ DIFFERENT: project_details_CAD.sql6. ⚠️ DIFFERENT: project_revenue_CAD.sql Total number of different files: 5Total number of missing files in DEV CONTAINER: 0Total number of missing files in PRD CONTAINER: 1