Managing Dataflow Gen2 across multiple environments (Development and Production) can be a headache. By default, the destination Lakehouse or Warehouse IDs are hardcoded. This means when you deploy to a Production workspace, your dataflow might still try to write back to your Development environment. In this guide, I’ll show you a workaround to parameterize your … Read more
What are Deployment Pipelines? Pipeline Structure Stage Comparison in UI Item Pairing & Status Selective Deployment Deployment Rules Lakehouse & Deployment Pipelines in Fabric Deployment pipelines move Lakehouse metadata, including shortcuts, but do not copy data or table schemas. After deployment, shortcuts still point to the original source, and the Lakehouse will be empty unless … Read more
When working with Git repositories in Azure DevOps, you might need to move a file from one folder to another without losing its history. The best way to do this is by using the git mv command. This ensures Git recognizes the move as a rename and keeps the file’s commit history intact. Why Not … Read more
This process explains how Data Engineering (DE) and Analytics Reporting (A) work together to turn a business need into a trusted, production-ready report. It’s designed for non-technical managers: no jargon, just what happens, who’s responsible, and what the outcomes are. Phase 0: Definition – Ground Zero PurposeLay the foundation before vision-shaping begins. Ensure clarity on … Read more
Step 1: Download SQL Files from Azure Storage Step 2: Flatten Directory Structure and Compare Files Using Python Notebook Step 3: Analyse Differences Using Visual Studio Code
Copy plugins/my/ includes/classes/fields_types
In Agile design, teams work in short sprints and prioritise the most important features to be developed first. Value-based prioritisation allows startups to get a Minimal Viable Product (MVP) to market quickly and start gathering feedback from customers. Additionally, and by definition, Agile design encourages safe-to-fail experimentation early on in the product’s lifecycle, giving the designer quick … Read more
The traditional design methods, which assume that developers fully understand the user’s requirements and that the user knows what they want before seeing the product, are often flawed in the real world. Complex systems, such as software products and their users, constantly evolve and adapt to users’ changing needs and preferences. As users interact with a software product, … Read more
A measure of good design is how dependent higher structures are on lower ones, and this is closely related to the principle of separation of concerns and cohesion. Cohesion is a measure of how closely the elements of a module or component work together to achieve a single, well-defined purpose. When a system is well-designed, the higher structures … Read more
Distributed large teams can face several challenges when producing Agile architecture. One of the main challenges is communication and coordination. Team members may be located in different locations and time zones and have different cultural backgrounds. These factors make having regular meetings, sharing information, and collaborating on design decisions difficult. Additionally, distributed teams may have difficulty building trust and developing a … Read more