Purpose: Transfer SharePoint list the Database environment for DataWarehouse Requires: Process Steps: 4. DataFlow Gen2 5. Power Query Window 6. Get Data 7. More… 8. SharepPoint Online List (NOT Sharepoint List) 9. Enter URL 10. Set Implementation to [2.0] 11. [Next] 12. Choose your list 13. [Create] 14 . Rename Dataflow 15. Select [Cog] on … Read more
Objective: The Snapshot pipeline aims to streamline data analysis and management by creating optimised views from raw data files. Process Overview: Loading parquet files and Delta lake table Creation: View Creation API Access and loading into the data warehouse Conclusion: The Snapshot pipeline offers an efficient solution for transforming raw data into actionable insights. By … Read more
Data pipeline name: My Bronze Pipeline Owner: Data Engineering Team Used since: July 2023 Purpose: The Bronze pipeline is an automated the loading of Parquet data files from an Azure data lake storage container into tables in the staging database Overview This pipeline orchestrates the movement of new data that is continually appended to storage. … Read more
Data pipeline name: Medallion to Bronze ETL Pipeline Owner: Data Engineering Team Used since: July 2023 Purpose: The Medallion pipeline is an automated process that handles the extraction, loading, and processing of CSV data from SATUK or GMISUK systems into bronze storage as parquet files. Overview The Medallion pipeline begins by setting source variables based … Read more