01 · Context
The challenge
A large raw banking transaction CSV needed to be ingested, cleaned, standardized, modelled and validated in a repeatable pipeline rather than processed manually in an exploratory notebook.
02 · Architecture
A clear path from raw to trusted.
The layers separate ingestion from cleaning and business-facing modelling, making each stage easier to reason about and validate.
03 · Engineering
Engineering approach
Bronze ingestion
Read the raw transaction file from a governed Volume and persist the source data as Delta.
Silver transformation
Handle missing values, required-field rules, identifiers and standardization before modelling.
Gold modelling
Create customer, employee, transaction and fact tables from the cleaned dataset.
Quality validation
Fail the workflow when the fact table is empty or required identifiers are missing.
04 · Orchestration
Dependency-based workflow.
ingestion.py
transformation.py
gold_transformations.py
data_quality.py
Each task runs only after its upstream dependency succeeds. Databricks provides run status, metrics and lineage around the code.
05 · Modelling
Gold data model
Customer
Customer-focused descriptive attributes.
Employee
Employee-related attributes.
Transaction
Transaction-level descriptive data.
Fact table
Keys and measures for analytics-ready querying.
06 · Reliability
Quality is part of the pipeline.
The final task validates that the fact table contains data and that customer, employee and transaction identifiers are not null. A failed rule raises an error so the workflow cannot silently report success with invalid output.
07 · Reflection
What I would improve next
- Parameterize catalog, schema and source paths for multiple environments.
- Move toward incremental ingestion when new source data arrives regularly.
- Expand quality checks to uniqueness, accepted values, referential integrity and freshness.
- Add monitoring and notifications for production-style operations.