Ali Ahmed Overview The myosh_Archive notebook is designed to maintain a complete and accurate archive of records from the MyOSH API. It ensures that no data is lost by identifying gaps in the current API response and backfilling missing records, then merging everything into a single, consolidated archive stored in the Azure Data Lake container. Why Do … Read more
Overview The cdm__Archive_upsert notebook is designed to help keep accurate historical records of data. It implements a technique called Slowly Changing Dimension Type 2 (SCD2), which means instead of overwriting old information, it creates a timeline of changes so you can see what was true at any point in time. This method ensures that previous … Read more
When working with Microsoft Fabric, one common challenge is keeping the SQL endpoint in sync with the Lakehouse endpoint after a pipeline run. Fabric provides two endpoints for every Lakehouse: Why does the SQL endpoint sometimes lag? The SQL endpoint doesn’t automatically refresh its metadata the moment new data lands in the Lakehouse. Instead, a … Read more
Today I stumbled upon a neat little trick in Power BI’s Power Query Editor, and it’s too good not to share! If you’ve ever loaded data into Power BI and ended up manually renaming columns one by one to clean them up, there’s a much faster way. 💡 You can rename all columns at once … Read more
Staging: Gl: Logging:
List all Error and Comment logging processes, that will be removed if we do not have an SQL server The current logging processes include transaction and error logs. These logs are stored in tables within the staging (stg) and Global Org (GL) databases. GL: Stg:
Data Source Identification: Data Ingestion and Storage: Data Staging: View Creation: Automated Deployment: Snapshot Creation and Storage: