Skip to content

Mini Resilient Data Platform

A local, production-style data platform that simulates e-commerce analytics end to end: Python and Faker generate events, an ETL pipeline validates and transforms them with error handling and retries, PostgreSQL stores the raw and modeled data, and Metabase serves the dashboards, all orchestrated with Docker Compose.

PythonPostgreSQLMetabaseDockerSQLFaker

Context

An end-to-end data platform that simulates e-commerce analytics, built to practise production data-engineering patterns on a single machine, without cloud infrastructure.

Problem

Demonstrate the full path from raw events to business dashboards, including the data-quality and reliability concerns real pipelines face, in a reproducible local setup.

My role

Sole author: designed the schema, the ETL and its quality checks, the SQL metric models, and the dashboards.

Architecture

Python with Faker generates realistic e-commerce events; an ETL pipeline validates, transforms, and loads them into PostgreSQL as raw and modeled tables; SQL builds the business metrics; Metabase serves executive dashboards; and Docker Compose orchestrates every service.

Key decisions

  • Chose PostgreSQL with SQL-based dimensional modeling over a heavier warehouse, to keep the whole stack reproducible locally while still showing real modeling.
  • Built retries and automated data-quality validation into the ETL so a bad batch fails loudly instead of silently corrupting downstream metrics.
  • Packaged the platform with Docker Compose so it spins up end to end with a single command.

Outcome

Runs end to end locally and renders live Metabase dashboards. The figures shown are from seeded, simulated data, not real-world business results.