Mini Resilient Data Platform
A local, production-style data platform that simulates e-commerce analytics end to end: Python and Faker generate events, an ETL pipeline validates and transforms them with error handling and retries, PostgreSQL stores the raw and modeled data, and Metabase serves the dashboards, all orchestrated with Docker Compose.
PythonPostgreSQLMetabaseDockerSQLFaker
Context
An end-to-end data platform that simulates e-commerce analytics, built to practise production data-engineering patterns on a single machine, without cloud infrastructure.
Problem
Demonstrate the full path from raw events to business dashboards, including the data-quality and reliability concerns real pipelines face, in a reproducible local setup.
My role
Sole author: designed the schema, the ETL and its quality checks, the SQL metric models, and the dashboards.
Architecture
Python with Faker generates realistic e-commerce events; an ETL pipeline validates, transforms, and loads them into PostgreSQL as raw and modeled tables; SQL builds the business metrics; Metabase serves executive dashboards; and Docker Compose orchestrates every service.
Key decisions
- Chose PostgreSQL with SQL-based dimensional modeling over a heavier warehouse, to keep the whole stack reproducible locally while still showing real modeling.
- Built retries and automated data-quality validation into the ETL so a bad batch fails loudly instead of silently corrupting downstream metrics.
- Packaged the platform with Docker Compose so it spins up end to end with a single command.
Outcome
Runs end to end locally and renders live Metabase dashboards. The figures shown are from seeded, simulated data, not real-world business results.