LakeForge

LakeForge is a local, production-grade Modern Data Engineering Platform that powers both analytics and AI. It is built incrementally, phase by phase, using the same technologies and patterns you would find in a real enterprise data team.

Repo path: lakeforge · Python package: data_platform (internal library name, unchanged)

The primary learning objective is Data Engineering (~80%). AI (~20%) is treated purely as a consumer of the platform.

What we are building

An end-to-end platform that can:

Ingest structured and unstructured enterprise data
Process data in batch and via streaming
Store data in a Lakehouse using the medallion architecture (Bronze / Silver / Gold)
Transform data into business models with dbt
Expose data for analytics (APIs + dashboards)
Expose data for AI (a simple RAG application)

Everything runs locally via Docker Compose. No cloud services.

Technology stack

Concern	Technology	Introduced in
Language	Python	Phase 0
Operational DB / warehouse	PostgreSQL	Phase 0 / 1
Object storage (data lake)	MinIO (S3-compatible)	Phase 0 / 2
Orchestration	Apache Airflow	Phase 3
Distributed processing	Apache Spark (PySpark)	Phase 4
Lakehouse table format	Apache Iceberg	Phase 5
Transformation / modeling	dbt Core	Phase 6
Streaming	Apache Kafka	Phase 7
API	FastAPI	Phase 8
Analytics	Dashboards	Phase 9
AI	fastembed + Qdrant + RAG (local)	Phase 10
Containerization	Docker + Docker Compose	Phase 0
Testing	pytest	Phase 0

Architecture (target)

flowchart LR
    subgraph sources [Data Sources]
        postgres[(PostgreSQL)]
        rest_api[REST APIs]
        files[CSV / JSON / PDF]
    end
    subgraph ingestion [Ingestion]
        airflow[Airflow]
        kafka[Kafka]
    end
    minio[(MinIO / Raw)]
    spark[Spark]
    subgraph lake [Lakehouse - Iceberg]
        bronze[(Bronze)]
        silver[(Silver)]
        gold[(Gold)]
    end
    dbt[dbt Core]
    subgraph serving [Serving]
        api[FastAPI]
        dash[Dashboards]
        rag[RAG]
    end

    postgres & rest_api & files --> airflow --> minio
    kafka --> minio
    minio --> spark --> bronze --> silver
    silver --> dbt --> gold
    gold --> api & dash & rag

See architecture/ for per-phase diagrams.

Prerequisites

Docker Desktop running
Python 3.9+
~8 GB free RAM
Ports available: 5432, 8000, 8080, 8501, 9000, 9001, 9092, 6333

Step-by-step build & verify (Phases 0–10)

Copy-paste each phase in order. After every phase, run the Verify block before moving on. For more detail (troubleshooting, tear-down), see docs/RUNBOOK.md. Latest E2E results: docs/VERIFICATION-REPORT.md.

Bootstrap (once)

cp .env.example .env
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,streaming,api]"
docker info    # Docker daemon must respond
pytest -m unit -q

Moved or renamed the repo folder? Delete .venv and rerun the block above. Virtualenv scripts store absolute paths and break after a move (bad interpreter).

Phase 0 — Core infrastructure

./scripts/dev-setup.sh up

Verify:

Check	Command / URL	Expected
Postgres	`docker ps \| grep dp_postgres`	`healthy`
MinIO	`docker ps \| grep dp_minio`	`healthy`
Buckets	http://localhost:9001 (`minioadmin` / `minioadmin`)	`raw`, `bronze`, `silver`, `gold`, `warehouse`

Phase 1 — Source data (PostgreSQL)

python -m data_platform.sources.seed

Verify:

docker exec dp_postgres psql -U platform -d platform -c "\dt ecommerce.*"
docker exec dp_postgres psql -U platform -d platform -c \
  "SELECT 'customers' t, count(*) FROM ecommerce.customers
   UNION ALL SELECT 'products', count(*) FROM ecommerce.products
   UNION ALL SELECT 'orders', count(*) FROM ecommerce.orders
   UNION ALL SELECT 'order_items', count(*) FROM ecommerce.order_items;"

Expected: 500 / 120 / 2000 / ~5857 rows (schema is ecommerce, not public).

UI: DBeaver → localhost:5432, db platform, user platform, pass platform_pass → schema ecommerce.

Phase 2 — Raw landing (MinIO)

python -m data_platform.ingestion.extract_raw

Verify: MinIO console http://localhost:9001 → bucket raw → folders postgres_ecommerce/, currency_api/, files/, github/, jira/ with ingest_date=YYYY-MM-DD/ partitions.

Phase 3 — Airflow orchestration

./scripts/dev-setup.sh airflow
# wait ~60s for the webserver

Verify:

UI: http://localhost:8080 — login admin / admin
DAG raw_ingestion visible and unpaused; trigger manually → all tasks green

docker exec dp_airflow_scheduler airflow dags list | grep raw_ingestion

Phase 4 — Bronze (Spark Parquet)

./scripts/dev-setup.sh spark
export INGEST_DATE=$(date -u +%F)
docker exec dp_spark /opt/spark/bin/spark-submit --master 'local[*]' \
    /opt/app/spark/jobs/build_bronze.py --ingest-date "$INGEST_DATE"

Verify:

MinIO → bucket bronze → partitioned Parquet under postgres_ecommerce/, files/
Spark UI: http://localhost:4040 — only live while spark-submit is running; open it in another tab during the job above. After the job exits, :4040 goes blank (that is normal).

Phase 5 — Silver (Iceberg)

First time only — create the Iceberg catalog database:

docker exec dp_postgres psql -U platform -d postgres -tc \
  "SELECT 1 FROM pg_database WHERE datname='iceberg'" | grep -q 1 || \
  docker exec dp_postgres psql -U platform -d postgres -c "CREATE DATABASE iceberg;"

Build Silver:

docker exec dp_spark /opt/spark/bin/spark-submit --master 'local[*]' \
    /opt/app/spark/jobs/build_silver.py --ingest-date "$INGEST_DATE"

Verify:

docker exec -w /tmp dp_spark /opt/spark/bin/spark-sql -e \
  "SHOW TABLES IN lakehouse.silver; SELECT count(*) FROM lakehouse.silver.orders;"

Expected: 4 Silver tables; 2000 orders. MinIO → warehouse/silver/ (Iceberg data/ + metadata/).

Phase 6 — Gold (dbt star schema)

./scripts/dev-setup.sh dbt

# First time only — namespaces the Thrift Server expects
docker exec -w /tmp dp_spark /opt/spark/bin/spark-sql -e \
  "CREATE NAMESPACE IF NOT EXISTS lakehouse.default; CREATE NAMESPACE IF NOT EXISTS lakehouse.gold;"

docker exec dp_dbt dbt debug
docker exec dp_dbt dbt build

Verify: 4 mart models built; 18/18 dbt tests PASS. MinIO → warehouse/gold/.

docker exec dp_dbt dbt test

Phase 7 — Kafka streaming

./scripts/dev-setup.sh kafka
python -m data_platform.streaming.producer --count 2000
docker exec dp_spark /opt/spark/bin/spark-submit --master 'local[*]' \
    /opt/app/spark/jobs/stream_clickstream_bronze.py

Verify:

docker exec -w /tmp dp_spark /opt/spark/bin/spark-sql -e \
  "SELECT count(*) FROM lakehouse.bronze.clickstream_events;"

Expected: ~2000 rows.

Phase 8 — Data API (FastAPI)

./scripts/dev-setup.sh api

Verify:

curl -s http://localhost:8000/health | python3 -m json.tool
curl -s http://localhost:8000/analytics/revenue-by-category | python3 -m json.tool
curl -s http://localhost:8000/analytics/funnel | python3 -m json.tool

UI: http://localhost:8000/docs (Swagger)

Expected: /health → "tables_reachable": true; revenue returns 6 categories.

Phase 9 — Analytics dashboard (Streamlit)

./scripts/dev-setup.sh dashboard

Verify:

UI: http://localhost:8501
KPI row, revenue chart, top products, clickstream funnel, recent events table all render

curl -s -o /dev/null -w "Streamlit: HTTP %{http_code}\n" http://localhost:8501/_stcore/health

Expected: HTTP 200.

Phase 10 — AI / RAG (fastembed + Qdrant)

./scripts/dev-setup.sh ai
# first run downloads BAAI/bge-small-en-v1.5 (~30s) and indexes 128 documents

Verify:

open "http://localhost:8000/ai/ask?q=what%20is%20the%20return%20policy"
curl -s "http://localhost:8000/ai/search?q=return%20policy&top_k=3" | python3 -m json.tool
curl -s "http://localhost:8000/ai/ask?q=what%20is%20the%20return%20policy" | python3 -m json.tool

UI:

RAG ask: http://localhost:8000/ai/ask?q=what%20is%20the%20return%20policy
Qdrant: http://localhost:6333/dashboard → collection knowledge_base (128 points)

Expected: top search hit "Return & Refund Policy"; ask returns 30-day refund text.

Final automated checks

pytest -q           # expect 64 passed, 1 skipped
ruff check .
docker exec dp_dbt dbt test   # 18/18 PASS

All UIs at a glance

Service	URL	Credentials
MinIO console	http://localhost:9001	`minioadmin` / `minioadmin`
PostgreSQL (DBeaver)	`localhost:5432`, schema `ecommerce`	`platform` / `platform_pass`
Airflow	http://localhost:8080	`admin` / `admin`
Spark UI (during jobs)	http://localhost:4040	—
Data API (Swagger)	http://localhost:8000/docs	—
Analytics dashboard	http://localhost:8501	—
RAG ask (browser)	http://localhost:8000/ai/ask?q=what%20is%20the%20return%20policy	—
Qdrant dashboard	http://localhost:6333/dashboard	—

Tear down: ./scripts/dev-setup.sh down (keep data) · ./scripts/dev-setup.sh reset (wipe volumes)

Docs: RUNBOOK · Verification report · Advanced roadmap (planning only)

Repository layout

Folder	Purpose
`docs/`	Concept guides and per-phase lessons
`architecture/`	Architecture & data model diagrams
`src/data_platform/`	Shared Python library (config, connections)
`airflow/`	DAGs and orchestration
`spark/`	PySpark processing jobs
`dbt/`	dbt Core project (Gold models)
`api/`	FastAPI serving layer
`storage/`	Object storage setup & policies
`kafka/`	Streaming producers/consumers
`tests/`	Unit + integration tests
`docker/`	Dockerfiles & compose overlays
`scripts/`	Setup & operational scripts
`notebooks/`	Exploratory notebooks
`sample-data/`	Generated sample datasets

Phases

Phase	Topic	Status
0	Project setup: Docker, Git, structure, docs	✅ Complete
1	PostgreSQL sample data & generation	✅ Complete
2	MinIO object storage & raw data	✅ Complete
3	Airflow ETL orchestration	✅ Complete
4	Spark transformations & partitioning (Bronze)	✅ Complete
5	Apache Iceberg lakehouse (Silver)	✅ Complete
6	dbt business models (Gold star schema)	✅ Complete
7	Apache Kafka streaming ingestion	✅ Complete
8	FastAPI serving layer (analytics API)	✅ Complete
9	Analytics dashboard (Streamlit)	✅ Complete
10	AI: embeddings, Qdrant, RAG	✅ Complete

Each phase lesson lives in docs/. Start with docs/phase-00-project-setup.md.

For end-to-end recreation see docs/RUNBOOK.md. For the latest verification results see docs/VERIFICATION-REPORT.md. For advanced/production topics see docs/ADVANCED-ROADMAP.md.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

LakeForge

What we are building

Technology stack

Architecture (target)

Prerequisites

Step-by-step build & verify (Phases 0–10)

Bootstrap (once)

Phase 0 — Core infrastructure

Phase 1 — Source data (PostgreSQL)

Phase 2 — Raw landing (MinIO)

Phase 3 — Airflow orchestration

Phase 4 — Bronze (Spark Parquet)

Phase 5 — Silver (Iceberg)

Phase 6 — Gold (dbt star schema)

Phase 7 — Kafka streaming

Phase 8 — Data API (FastAPI)

Phase 9 — Analytics dashboard (Streamlit)

Phase 10 — AI / RAG (fastembed + Qdrant)

Final automated checks

All UIs at a glance

Repository layout

Phases

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 14 Commits
airflow		airflow
api		api
architecture		architecture
dbt		dbt
docker		docker
docs		docs
kafka		kafka
notebooks		notebooks
sample-data		sample-data
scripts		scripts
spark		spark
src/data_platform		src/data_platform
storage		storage
tests		tests
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

LakeForge

What we are building

Technology stack

Architecture (target)

Prerequisites

Step-by-step build & verify (Phases 0–10)

Bootstrap (once)

Phase 0 — Core infrastructure

Phase 1 — Source data (PostgreSQL)

Phase 2 — Raw landing (MinIO)

Phase 3 — Airflow orchestration

Phase 4 — Bronze (Spark Parquet)

Phase 5 — Silver (Iceberg)

Phase 6 — Gold (dbt star schema)

Phase 7 — Kafka streaming

Phase 8 — Data API (FastAPI)

Phase 9 — Analytics dashboard (Streamlit)

Phase 10 — AI / RAG (fastembed + Qdrant)

Final automated checks

All UIs at a glance

Repository layout

Phases

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages