DE304: Cloud & DevOps Practices in Data Engineering

by

SFIA Reference: METL Level 4/5

Data Engineer (Grade 3)

Competency Description

Implements automated, repeatable DevOps practices and infrastructure as code in data engineering.

Learning Outcomes

  • Create CI/CD pipelines using Azure DevOps or GitHub Actions.
  • Use Terraform/Bicep for Infrastructure as Code.
  • Maintain version control for scripts, transformations, and metadata.

Evidence Requirements

Level 4:

  • Deployed a data pipeline via CI/CD.
  • Versioned data transformation scripts in Git.

Level 5:

  • Standardised CI/CD templates across team.
  • Implemented IaC for multiple environments.

Suggested Learning Activities

  • Complete a DevOps bootcamp (internal or external).
  • Rebuild a legacy SQL-based flow into CI/CD pipeline.
  • Conduct a peer DevOps audit.

Recommended Resources

  • GitHub Actions for Data Engineers (KnowHow)
  • Terraform Azure Provider Docs
  • Bicep and IaC Examples (Internal)

Leave a Comment