A Week in the Life of a Data Engineer

The Data Engineering Team plays a crucial role in ensuring the smooth operation and continuous improvement of our data services. However, many people may not fully understand how we structure our development cycles and why we operate in a particular way. This article aims to provide insight into a typical week in the life of a data engineer, illustrating when we develop, deliver, and ensure quality.

Monday – Delivery Day

Monday marks the culmination of our weekly cycle – it’s Delivery Day. On this day, we release updates to the data warehouse, new reports, and enhancements. We deliberately choose Monday because delivering on a Friday comes with significant risks:

Delayed Visibility: If we send an update on a Friday, most stakeholders will only see it on Monday anyway, making Friday an inefficient day for major releases.
Support Gaps: While our releases are well-tested, real-world usage can surface unexpected issues. If a problem arises on a Friday, few people are available to address it over the weekend, potentially leading to delays in fixes.

By delivering on Mondays, we ensure that any unforeseen issues can be quickly identified and resolved with the full team on hand.

Tuesday – Task List Review & Release Validation

Tuesdays are dedicated to assessing priorities and planning upcoming work. In our Task List Review meeting, we:

Collect all outstanding requests, bug reports, and promised developments.
Discuss these items with the data warehouse stakeholders.
Prioritise the work for the upcoming development cycle.
Define how tasks will be released and tested/accepted.

Tuesday is also the day when the previous release can be fully observed in the data, as changes deployed on Monday often contain code activated during the overnight import process. This means that Tuesday provides the first opportunity to review how the new release is performing in production.

This structured approach ensures alignment between business needs and development efforts.

Wednesday & Thursday – Development Days

These two days are dedicated to focused development. We break down tasks so that they can be completed within one and a half days, allowing time for:

Coding and implementation.
Discussion with stakeholders to refine and confirm requirements.
Initial testing to ensure quality before release preparation.

A follow-up meeting on Thursday usually allows us to validate that what we are building aligns with expectations, making adjustments if necessary before finalising the work.

Funky Friday – Documentation & Innovation

Funky Friday is our chance to step back and ensure all our work is well-documented and fully integrated into our knowledge-sharing platforms, such as KnowHow. We dedicate time to:

Writing and refining documentation.
Ensuring all code is fully deployed in the development environment.
Working on innovative side projects or process improvements.

By setting aside this time, we ensure that we always have space for professional development, process enhancements, and maintaining high-quality documentation. Over a year, this equates to 50 hours of dedicated innovation time, allowing us to explore new ideas that may not fit within standard development cycles.

A Predictable, Agile Development Cycle

By following this structured weekly cycle, we ensure:

Regular releases with predictable updates.
Short, well-packaged development periods that allow for agility and responsiveness.
Clear expectations for when stakeholders can expect changes to their data marts, reports, and services.

This approach enables us to deliver high-quality data solutions while maintaining flexibility and continuous improvement. Understanding this cycle allows our clients and stakeholders to know when to expect updates and how best to engage with the Data Engineering Team.