Data Sources and Integration

by

4.1 Data Sources

There are numerous data sources in BMT many are integrated together in some way, and some are large scale Enterprise Applications and others are smaller.  The Enterprise Applications within BMT is frequently under review as we rationalise and modernise our global systems. 

4.2 Integrations Strategy

The Integration Strategy prioritises seamless connections between Enterprise Applications and the Global Data Warehouse to ensure standardised and accurate reporting.

Key Objectives:

  • Centralised Integration: Systems such as GMIS and the Performance Portal (Cornerstone) connect directly to the Data Warehouse to align operational and reporting standards.
  • Enhanced Accessibility: Build a consistent and reliable data flow to facilitate cross-functional analysis.
  • Business Alignment: Ensure that integration supports overarching strategic goals and enhances decision-making capabilities.
  • The strategy is designed to support both real-time and batch data processing requirements, with scalability to accommodate future applications.

Data Sources and Integration Principle 1 – Centralised Integrations

All core and common Enterprise Applications required are integrated into a Global Data Warehouse for Business Reporting.  Regional reporting will be from the Global Data Warehouse to remove duplication and ensure consistency and cost-effective operation.

1.3 Integrations Architecture

The Integrations Architecture employs an API-First approach to achieve seamless, secure, and efficient data management.

Key Components:

  • API-First Approach: APIs are the preferred method for real-time integration with cloud and SaaS systems.
  • Secure Push Mechanisms: Regional and National systems, such as GMIS, may opt to, securely push data to the Global Data Warehouse.
  • Cloud-Native Infrastructure: M365 and on-premises systems utilise cloud-native solutions, such as Power Query and Fabric, for scalability and reliability.
  • Standardisation via Data Models: The Common Data Model (CDM) ensures all integrated systems follow a unified framework for data exchange.

Data Sources and Integration Principle 2 – API-First Integrations

Where available the information flows between Enterprise Applications will utilise API based flows rather than file transfer approaches. 

1.4 Data Synchronisations

Data synchronisation ensures timely and consistent updates across systems:

  • 24-Hour Cycles: Standard updates occur daily, ensuring accuracy without overloading systems.
  • Flexible Schedules: Certain regions or critical systems may adopt more frequent synchronisations (e.g., hourly updates for incremental loads) and utilise a conditional load approach to not cause performance issues for any particular region.[SW1] 
  • Strategic Scheduling: Tailored schedules (daily, weekly, monthly) minimise user impact while meeting business needs.
  • Incremental and Full Loads: Incremental updates are preferred where possible to optimise performance, while full loads may be scheduled for periodic data reconciliation.
Example IFS Data Load Approach

Data Sources and Integration Principle 3 – Synchronisation of Data

Data will be synchronised to meet the business requirement.  Approaches will seek to optimise the cost of synchronising data utilising incremental, conditional and scheduled loads of data to avoid excessive processing of data where it is not required.

1.5 Interoperability

Interoperability is driven by the adoption of the Common Data Model (CDM), ensuring consistency across diverse systems and functions.

Key Benefits:

  • Service-Oriented Architecture: CDM provides a framework to standardise data for Projects, Customers, Finance, and Employee systems.
  • Automated Interoperability: The model enables automated data flows, reducing manual intervention and enhancing efficiency.
  • Cross-System Consistency: Unified terminologies and formats simplify data sharing and integration across the organisation.
  • The integration framework is designed to support both current and future applications, ensuring agility and scalability for evolving business requirements.

 [SW1]Slight tweak here to look at conditional load as well as incremental load as a way of enabling “midnight” load for each region.

Leave a Comment