Staging:
- All tables in the stg databases (UK, CUS, APC) have been recreated as views in the dwh_bronze Synapse database, using OPENROWSET to read data directly from Parquet files from the bronze container.
- The snapshot pipeline has been changed so that the copy activity now executes the querys from the dwh_bronze Synapse on demand database. therefore eliminating the stg database from the process
Gl:
- GL stores the cdm tables, snp’s and runs get.myview
Logging:
- Logging occurs in the database where data is loaded. For example, snps’s that are loaded from the gold container to GL database are logged in the trans_mylog table in the GL database.
- When data moves from its source to the bronze container , logging happens in the trans_mylog table of their respective staging database. IFS App 10 data transactions are recorded in stg_uk, while SharePoint data from APAC is logged in stg_apac.