Assurance System Test, Regression Test, Unit Test activities, UA Tests, Peer Reviews (with estimated dates), code review methodology (Pull Requests/Pair Programming/other) A new DataMart feature is more than several lines of code. It is usually a multilayer, complex pipeline system, incorporating separate functional components and third-party data integrations. Therefore, efficient development testing should go far … Read more
SQP, requirements catalogue, package details, method of rolling out. DataMart Lifecycle (Development Plan) Output Purpose Owner Location Data Source Inventory Identify relevant data sources within the organisation, including databases, applications, and external sources. Data Architect/Engineer (with Data Specialist) myBMT(Data Objects) Data profiling reports Profile data sources to understand data quality, structure, and relationships. Data Architect/Engineer … Read more
Frequency, location and style of meetings, who should attend. To include standups/sprint meetings, code review methods and end of project review. Incorporating an Agile approach forms an integral aspect of the Data Engineering development process. By dividing the development cycle into smaller segments, iterations, and sprints, Agile methodologies enable engineers and data specialists to collaborate … Read more
People involved and their roles, chains of command and reporting methods, distribution list of SQP. Data Steering Committee Data Stewards Data Specialists Quality Behaviours of Data Specialists Data Engineering Data Mesh Principle The Data Mesh decentralises data ownership by transferring the responsibility from the central data team to the business units that create and consume … Read more
Scope covered by the project and any documents received. Technical Perspective The Data Warehouse Development Quality Plan describes the creation, organisation, movement, change and storage of data from the perspective of the technical and physical implementation. The scope of the Data Warehouse Development Quality Plan encompasses the following key areas: Business Perspective The business perspective … Read more
Security specific requirements for the project. Assessment of changes on existing system security. The confidentiality, integrity, and availability (CIA) model is widely used by organisations to implement appropriate security controls and policies, which helps identify key problem areas and the necessary solutions to resolve these issues. Access Control and Authorisation: Role-based access mechanisms and strict … Read more
Aim to completely transform the data infrastructure of the organisation, making it agile, scalable, and highly efficient. This could involve implementing cutting-edge technologies like distributed computing, real-time processing, and machine learning pipelines.
Set a goal to seamlessly integrate data from diverse sources and platforms within the organisation. This could involve building connectors, APIs, and data pipelines to facilitate smooth data flow between different systems and applications.
Make it a goal to foster a culture of continuous learning and innovation within the data engineering team. Encourage team members to stay updated on the latest technologies and methodologies, experiment with new ideas, and share their learning with the broader organisation.
Becoming a data-driven business doesn’t happen overnight. It takes time, money, and effort to develop the combination of knowledge, skills, and technological proficiency needed to weave business intelligence into the cultural fabric of your organisation. It’s a journey: one with many steps and milestones along the way. To reach your destination safely, you need a guide. … Read more