Overview
As a Senior Data Engineer (Grade 4) you design, build, and optimise scalable data platforms, pipelines, and architectures. In this role, you lead data engineering best practices, DevOps, automation, and real-time processing, working closely with Data Architects, Analysts, and Data Scientists to drive business value from data.
You play a key role in architecting cloud-based data solutions, ensuring performance, scalability, security, and governance, and mentoring junior engineers to build a strong engineering culture.
You are passionate about data architecture, big data processing, and DevOps for Data, this role offers an opportunity to shape and drive enterprise-wide data engineering initiatives.
Key Responsibilities
1. Data Architecture & Data Platform Design
(SFIA: Solution Architecture – ARCH Level 5)
- Lead the design and implementation of scalable data architectures to support analytics, BI, and AI/ML workloads.
- Define enterprise data engineering best practices, including data lakehouse, data warehouse, and data mesh strategies.
- Own the architecture and optimisation of data models, data lakes, and relational database structures.
- Define data partitioning, indexing, clustering, and caching strategies for big data processing.
- Evaluate new technologies to enhance performance, resilience, and cost-effectiveness.
2. Data Pipeline Engineering & Automation
(SFIA: Data Engineering – DENG Level 5)
- Develop and maintain high-performance, automated ETL/ELT pipelines for batch and real-time data ingestion.
- Implement data transformation and data quality frameworks that ensure accurate, validated, and audit-ready data.
- Optimise data ingestion, storage, and processing for structured and semi-structured data.
- Drive the adoption of event-driven architectures and streaming data pipelines (Kafka, Azure Event Hub, AWS Kinesis).
3. DevOps & Infrastructure as Code for Data Engineering
(SFIA: Methods & Tools – METL Level 5)
- Implement CI/CD pipelines for data solutions using GitHub Actions, Azure DevOps, Terraform, or CloudFormation.
- Automate infrastructure provisioning (IaC) for data lakes, warehouses, and pipeline infrastructure.
- Ensure version control and best practices for SQL, Python, and cloud-based data solutions.
- Build observability solutions using monitoring tools (Datadog, Azure Monitor, Prometheus, or OpenTelemetry).
4. Real-Time Data & API Integrations
(SFIA: Systems Integration – INCA Level 5)
- Develop real-time data streaming solutions using Kafka, Azure Event Hub, or AWS Kinesis.
- Build API-based data integrations for dynamic, near-real-time analytics.
- Design and implement change data capture (CDC) and incremental data processing strategies.
5. Data Governance, Security & Compliance
(SFIA: Information Security – SCAD Level 5)
- Define and enforce data security best practices, including encryption, RBAC, and ABAC controls.
- Ensure compliance with GDPR, ISO 27001, and other regulatory requirements.
- Establish metadata management, data lineage tracking, and data cataloguing solutions.
6. Performance Optimisation & Cost Management
(SFIA: Systems Development – DESN Level 5)
- Optimise query performance and database workloads, reducing cloud costs and processing time.
- Implement cost-efficient storage solutions using Azure Synapse, AWS Redshift, Snowflake, or Databricks.
- Design data retention and archival strategies that balance performance and cost efficiency.
7. Leading & Mentoring Data Engineering Teams
(SFIA: People Management – LEDR Level 5)
- Mentor and coach junior and mid-level Data Engineers, fostering a culture of technical excellence.
- Lead code reviews and architectural discussions, ensuring best practices and consistency.
- Act as a technical advisor to cross-functional teams, including BI, Data Science, and Software Engineering.
8. Agile Delivery & Continuous Improvement
(SFIA: Methods & Tools – METL Level 5)
- Work in Agile sprints, defining backlog prioritisation, technical debt reduction, and sprint planning.
- Drive continuous integration and automation in data engineering processes.
- Collaborate with stakeholders to improve data reliability, scalability, and analytics capabilities.
What You Bring…
✅ Technical Expertise
- Expertise in SQL & Python (& PHP), with experience optimising queries, stored procedures, and ETL/ELT processes.
- Strong experience with cloud data platforms (Azure Synapse, AWS Redshift, Snowflake, Databricks).
- Hands-on experience with CI/CD tools (GitHub Actions, Azure DevOps, Terraform, CloudFormation).
- Knowledge of streaming architectures (Kafka, Azure Event Hub, AWS Kinesis).
- Experience designing distributed computing solutions (Spark, Delta Lake, or Hadoop-based ecosystems).
✅ Data Architecture & Governance
- Deep understanding of data lakehouse, warehouse, and data mesh architectures.
- Experience implementing data governance, lineage tracking, and metadata management solutions.
- Strong background in data security and compliance (RBAC, ABAC, encryption, GDPR, ISO 27001).
✅ Cloud & DevOps Mindset
- Proven experience deploying data solutions in cloud environments (Azure, AWS, GCP).
- Strong understanding of cost-optimisation strategies for cloud data storage and compute.
- Hands-on experience with Infrastructure as Code (Terraform, ARM Templates, Bicep, CloudFormation).
✅ Leadership & Collaboration
- Experience mentoring junior and mid-level Data Engineers and leading engineering discussions.
- Strong communication skills, capable of explaining complex technical concepts to non-technical stakeholders.
- Experience working in Agile environments (Scrum, Kanban) and leading data engineering initiatives.
✅ Desirable Experience
- Experience working with Power BI datasets, DAX calculations, and semantic models.
- Exposure to machine learning and MLOps for feature engineering and model deployment.
- Knowledge of containerisation (Docker, Kubernetes) for scalable data solutions.
Why Us?
🌍 Architect cutting-edge cloud data platforms for a global organisation.
🚀 Lead enterprise-wide data engineering initiatives and mentor junior engineers.
🔄 Drive DevOps & CI/CD adoption for scalable and automated data solutions.
📊 Work on real-time streaming, AI-driven analytics, and large-scale data processing.
💡 Innovate with modern cloud data technologies, including Azure Synapse, Snowflake, and Databricks.
🔑 Flexible hybrid working (2 days in-office per week).