Managing Dataflow Gen2 across multiple environments (Development and Production) can be a headache. By default, the destination Lakehouse or Warehouse IDs are hardcoded. This means when you deploy to a Production workspace, your dataflow might still try to write back to your Development environment. In this guide, I’ll show you a workaround to parameterize your … Read more
Delta tables are a core feature of the Microsoft Fabric Lakehouse. They solve a lot of the problems you normally hit with data lakes, especially around trust, auditing, and knowing what actually happened to your data. At a basic level, Delta tables give you ACID transactions on top of lake storage, but the real value … Read more
What are Deployment Pipelines? Pipeline Structure Stage Comparison in UI Item Pairing & Status Selective Deployment Deployment Rules Lakehouse & Deployment Pipelines in Fabric Deployment pipelines move Lakehouse metadata, including shortcuts, but do not copy data or table schemas. After deployment, shortcuts still point to the original source, and the Lakehouse will be empty unless … Read more
Overview The cdm_Archive_to_STG notebook is a critical component in the cdm_today and cdm_Archive pipeline. Its primary role is to create a staging table that represents a point-in-time snapshot of source data, which is then used by the cdm__Archive_upsert notebook to accurately update the main dimension table. This staging layer acts as a buffer between raw … Read more
Overview The cdm__Archive_upsert notebook is designed to help keep accurate historical records of data. It implements a technique called Slowly Changing Dimension Type 2 (SCD2), which means instead of overwriting old information, it creates a timeline of changes so you can see what was true at any point in time. This method ensures that previous … Read more
When working with Microsoft Fabric, one common challenge is keeping the SQL endpoint in sync with the Lakehouse endpoint after a pipeline run. Fabric provides two endpoints for every Lakehouse: Why does the SQL endpoint sometimes lag? The SQL endpoint doesn’t automatically refresh its metadata the moment new data lands in the Lakehouse. Instead, a … Read more