Month: March 2024

An Overview of the Snapshot Pipeline

by

Objective: The Snapshot pipeline aims to streamline data analysis and management by creating optimised views from raw data files. Process Overview: Loading parquet files and Delta lake table Creation: View Creation API Access and loading into the data warehouse Conclusion: The Snapshot pipeline offers an efficient solution for transforming raw data into actionable insights. By … Read more

Resource.Details

by

Purpose The Common View for Resource Details, excluding dates or measures, acts as a unified and reliable source for data on project resources. It is accessible to different departments and systems across the organisation. This view supports integration with fact tables for reporting and analysis and provides a limited set of dimension attributes. Use Case … Read more

Supplier.Invoice

by

Purpose The Supplier Invoice (value) fact table is designed to display quantitative data related to invoices generated. This table integrates with dimension tables, such as supplier_details, to provide detailed insights into the financial aspects of projects, including invoicing, revenue, and billing trends. The primary purpose is to support financial analysis and reporting across the organisation. Use … Read more

Transforming Your Data with Python: CSV to Parquet Conversion and NaN Handling

by

Efficient data storage and processing are crucial for businesses and organizations dealing with large datasets. Apache Parquet is a popular columnar storage format offering fast query performance and data compression, while CSV is a row-based format that may not be suitable for large-scale processing. This blog post covers how to convert CSV files to Parquet … Read more

Pandas.merge() function

by

The merge() function in Pandas is a powerful tool for combining two or more dataframes based on one or more keys. It is analogous to the JOIN operation in SQL databases and offers various options to customize the merge behavior. Here’s the basic syntax of the merge() function: pandas.merge(left, right, how=’inner’, on=None, left_on=None, right_on=None, left_index=False, … Read more

Customer.Invoice

by

Purpose Customer Invoice, provides high-level invoice details primarily focused on essential customer and financial information. It is structured to include invoice metadata, billing details, amounts, and various identifiers. The view originates from multiple snapshots and iterations, with each change tracked meticulously to support functionality and flexibility for different reporting and integration requirements. Use Case Owner: Soon … Read more

Customer.OrderLine

by

Purpose The Customer Order Line view provides essential details for individual order lines within customer orders. It serves as a granular component, detailing each line item’s specifics, such as quantity, pricing, revenue, dates, and relevant codes. Designed to align closely with the Customer Order view, this view enables precise tracking of each line item in … Read more