SFIA Reference: DENG Level 4/5
Competency Description
Designs, develops, and maintains data pipelines, ensuring scalability, performance, and quality in support of enterprise reporting and analytics.
Learning Outcomes
- Build automated pipelines for batch and semi-structured data.
- Apply transformation and validation logic using tools like ADF, Databricks.
- Optimise for performance and reusability across layers.
Evidence Requirements
Level 4:
- Independently delivered an end-to-end pipeline with monitoring and logging.
- Ensured data quality via reconciliation or validation steps.
Level 5:
- Designed scalable pipeline patterns adopted team-wide.
- Provided guidance or mentorship to junior engineers.
- Reviewed and refactored existing pipelines for efficiency or maintainability.
Suggested Learning Activities
- Complete Microsoft DP-203 certification.
- Internal lab: Build a Bronze-to-Gold ETL pipeline with quality checks.
- Participate in a pipeline design review and document findings.
- Case study reflection on a high-complexity ETL pipeline project.
Recommended Resources
- Microsoft Learn: Data Engineering on Azure
- Internal docs: ETL Framework Standards, ADF/Databricks Patterns
- KnowHow: “get.myhistory – Quality-controlled ETL practices”