Release Notes

Release 28

by

Alterations Busopp.Details_Plus (&202407) Remove CONVERT(NVARCHAR(32),HashBytes(‘MD5’, CONCAT_WS(‘|’,src.COMPANY,src.OPPORTUNITY_NO)),2) as busopp_item_id, as this is now generated in get.myView Created dvw.customer_item_202406 as part of removing Company reference from customer (& Supplier) Customer.Details Removed reference to COMPANY (and the associated check of customer.invoice) Customer (and Supplier) COMPANY defaults to 9900 Check for Alterations to Customer.InvoiceCutomer.OrderlineProject.CustomerInvoiceProject.CustomerOrderlineProject.ActivityDetails get.myView COMPANY now included as … Read more

Release 27

by

Released immediately due to data error discovered in [QFACT_BMT_ACTUAL_HOURS_BY_IAS$] PRD 20240731 Alterations Employee.Hours Change UPLIFT to CAST(GROSS AS DECIMAL(10, 3)) – CAST(NETT AS DECIMAL(10, 3)) AS UPLIFT, As UPLIFT was being delivered with discrepancies between rounding of GROSS and NETT i.e. 21.8104 – 21.81 = -4.0E-4 This was causing an error in both CAST and … Read more

SharePointList To Staging Pipeline

by

Purpose: Transfer SharePoint List data through azure blob storage and into the DataWarehouse Requires: Prerequisites: Process Steps: 4. Data Pipeline 5. Create pipeline parameters. Create two pipeline parameters. 5. Activities Tab 6. Look up Activity Click on the Lookup activity and go to settings, from there if not already set up create a new connection … Read more

DataSource.SharepointList

by

Purpose: Transfer SharePoint list the Database environment for DataWarehouse Requires: Process Steps: 4. DataFlow Gen2 5. Power Query Window 6. Get Data 7. More… 8. SharepPoint Online List (NOT Sharepoint List) 9. Enter URL 10. Set Implementation to [2.0] 11. [Next] 12. Choose your list 13. [Create] 14 . Rename Dataflow 15. Select [Cog] on … Read more

Release 25

by

Notes Alterations project_customerorderlineproject_customerorderline_202407 ADD PRE_ACCOUNTING_IDAwaiting arrival of [QFACT_PRE_ACCOUNTING_OL$]Internal Notes project_timeproject_time_202407 ADD INTERNAL_COMMENTsParallel change with project.expenses Resource_Activity_202407 Remove Resource Title, to use DataView (dvw) Additions project_activitydetailsproject_activitydetails_202407 [FACT_PROJ_ACTIVITY_INFO_OL] unique based on ACTIVITY_SEQ, CONTROL_CATEGORY_KEYThis table plans to be unique for ACTIVITY_SEQ only employee.leaveentitlementemployee.leaveentitlement_202406 myBMT | CR354: Employee Leave Entitlements (bmt-dwh-uks-app-my.azurewebsites.net) get.myView2 New version of get.myView for UAT Auto … Read more

An Overview of the Snapshot Pipeline

by

Objective: The Snapshot pipeline aims to streamline data analysis and management by creating optimised views from raw data files. Process Overview: Loading parquet files and Delta lake table Creation: View Creation API Access and loading into the data warehouse Conclusion: The Snapshot pipeline offers an efficient solution for transforming raw data into actionable insights. By … Read more

An Overview of the Bronze Pipeline

by

Data pipeline name: My Bronze Pipeline Owner: Data Engineering Team Used since: July 2023 Purpose: The Bronze pipeline is an automated the loading of Parquet data files from an Azure data lake storage container into tables in the staging database Overview This pipeline orchestrates the movement of new data that is continually appended to storage. … Read more

An Overview of the Medallion Pipeline

by

Data pipeline name: Medallion to Bronze ETL Pipeline Owner: Data Engineering Team Used since: July 2023 Purpose: The Medallion pipeline is an automated process that handles the extraction, loading, and processing of CSV data from SATUK or GMISUK systems into bronze storage as parquet files. Overview The Medallion pipeline begins by setting source variables based … Read more