ML Platform Shared Infrastructure

A reusable local infrastructure stack for applied ML engineering projects. It provides everything a modern ML system needs from a relational store, low-latency cache, event streaming, object storage, and observability — so you can focus on building ML components instead of wiring up services from scratch.

Start this stack before working on any project that needs databases, object storage, streaming, metrics, or dashboards.

Status

Verified on Windows with Docker Desktop:

.\scripts\setup.ps1
.\scripts\check.ps1

Expected result:

All checks passed.

Services

Service	URL / Host	Credentials	Used For
PostgreSQL	localhost:5432	postgres / postgres	Offline feature store, metadata, lineage, job state
Redis	localhost:6379	none	Online feature serving and low-latency cache
Redpanda / Kafka	localhost:9092	none	Event streams for ingestion, anomalies, feedback loops
Redpanda Console	http://localhost:8080	none	Kafka topic and message inspection
Schema Registry	http://localhost:8081	none	Kafka schema registry
PandaProxy	http://localhost:8082	none	Redpanda HTTP proxy
MinIO API	http://localhost:9000	minioadmin / minioadmin	S3-compatible artifact and dataset storage
MinIO Console	http://localhost:9001	minioadmin / minioadmin	Object storage browser
Prometheus	http://localhost:9090	none	Metrics collection
Grafana	http://localhost:3000	admin / admin	Dashboards and visual monitoring

Layout

ml-platform-infra/
|-- docker-compose.yml
|-- monitoring/
|   |-- prometheus.yml
|   `-- grafana/
|       |-- datasources.yml
|       |-- dashboards.yml
|       `-- dashboards/
|           `-- platform-overview.json
|-- scripts/
|   |-- check.ps1
|   |-- check.sh
|   |-- setup.ps1
|   |-- setup.sh
|   |-- teardown.ps1
|   `-- teardown.sh
|-- .env.example
|-- .gitignore
`-- README.md

Prerequisites

• Docker Desktop installed and running.
• Docker Compose v2 available through docker compose.

Clone

git clone https://github.com/Emart29/ml-platform-infra.git
cd ml-platform-infra

Start

From the repo root:

.\scripts\setup.ps1

Or from Bash / WSL / Git Bash:

./scripts/setup.sh

The setup script starts all services and prints docker compose ps so you can see the current status.

After startup, open:

• Grafana: http://localhost:3000
• Prometheus: http://localhost:9090
• Redpanda Console: http://localhost:8080
• MinIO Console: http://localhost:9001

Configure

The default ports and credentials are ready for local development. If a port is already in use, copy .env.example to .env and change only the values you need:

Copy-Item .env.example .env

cp .env.example .env

The .env file is ignored by git so local credentials and port overrides do not leak into the repo.

Keep the defaults unless you have a port conflict. Downstream projects can assume this stack is running with the default values.

Stop

Stop services and keep persistent data:

.\scripts\teardown.ps1

./scripts/teardown.sh

Stop services and delete persistent volumes:

.\scripts\teardown.ps1 --volumes

./scripts/teardown.sh --volumes

Quick Checks

Run the bundled health check:

.\scripts\check.ps1

./scripts/check.sh

Or inspect services manually:

docker compose ps
docker compose logs redpanda
docker compose logs prometheus

Prometheus should scrape itself and Redpanda. Grafana automatically provisions the Prometheus datasource and the ML Platform Overview dashboard.

Connection Strings

PostgreSQL: postgresql://postgres:postgres@localhost:5432/ml_platform
Redis: redis://localhost:6379/0
Kafka bootstrap server: localhost:9092
MinIO endpoint: http://localhost:9000
MinIO access key: minioadmin
MinIO secret key: minioadmin

Troubleshooting

If containers do not start, confirm Docker Desktop is running:

docker info

If a port is already in use, copy .env.example to .env and change the conflicting host port.

If a service starts but fails health checks, inspect its logs:

docker compose logs <service>

Examples:

docker compose logs redpanda
docker compose logs postgres
docker compose logs grafana

To fully reset local state:

.\scripts\teardown.ps1 --volumes
.\scripts\setup.ps1

This deletes persisted databases, Kafka data, MinIO objects, Prometheus history, and Grafana state.

Usage in Your Projects

Any project that depends on this stack should list it as a prerequisite:

Prerequisites: clone ml-platform-infra and run ./scripts/setup.sh or .\scripts\setup.ps1 first.