Skip to content
BranchesTags

Repository files navigation

PayFlow — Payment Gateway API

PayFlow is a production-style payment gateway API built in Go that simulates how fintech systems process bank payments. It handles the full transaction lifecycle: accepting payments, communicating with bank clients, tracking state, and notifying merchants with the reliability and auditability that financial systems require.

Designed to demonstrate production-oriented backend architecture: clean layering, distributed systems patterns, and the operational rigor expected in regulated environments.

Tech Highlights

  • Hexagonal architecture (ports and adapters)
  • Transactional outbox pattern (Kafka)
  • Circuit breaker + retry with exponential backoff and jitter
  • Idempotent payment processing (app + DB UNIQUE constraint)
  • State machine with full audit trail
  • Structured JSON logging (slog) with request ID correlation
  • Prometheus metrics — application (http_requests_total, http_request_duration_seconds, http_active_requests) + Go runtime; exposed at GET /metrics
  • OpenTelemetry distributed tracing (OTLP export, no-op when disabled)
  • Dockerized deployment

Architecture

flowchart TD
    LB["Load Balancer"]

    subgraph API["PayFlow API (stateless replicas)"]
        API1["Instance 1"]
        API2["Instance 2"]
        API3["Instance 3"]
    end

    PG[("PostgreSQL<br/>primary")]
    KAFKA[("Kafka<br/>events")]
    REDIS[("Redis*<br/>rate limit / idempotency")]

    WEBHOOK["Webhook Service*"]
    AUDIT["Audit Logger*"]
    ANALYTICS["Analytics Pipeline*"]

    LB --> API1 & API2 & API3
    API1 & API2 & API3 --> PG
    API1 & API2 & API3 --> KAFKA
    API1 & API2 & API3 --> REDIS
    KAFKA --> WEBHOOK & AUDIT & ANALYTICS
Loading

* Redis, webhooks, and analytics are shown for production reference. This repository implements the API, PostgreSQL persistence, Kafka-backed outbox flow, and Redis-backed rate limiting/idempotency coordination.

Payment Flow

sequenceDiagram
    participant Client
    participant API as PayFlow API
    participant DB as PostgreSQL
    participant Bank
    participant Kafka
    participant Downstream as Webhooks / Analytics

    Client->>API: POST /transactions (Idempotency-Key)
    API->>API: Check idempotency (duplicate? return cached result)
    API->>API: Validate merchant + amount
    API->>DB: BEGIN — payment + audit event + outbox event
    DB-->>API: COMMIT
    API->>Bank: Charge (circuit breaker + retry w/ backoff)
    alt Approved
        Bank-->>API: Approved
        API->>DB: status = COMPLETED, credit balance
    else Declined / failure
        Bank-->>API: Declined
        API->>DB: status = FAILED
    end
    API-->>Client: Response
    Note over API,Kafka: Async, decoupled from request path
    API->>Kafka: Outbox worker publishes event
    Kafka->>Downstream: Notify (webhooks, analytics)
Loading

Core Features

Transaction Processing

  • State machine enforcement: PENDING → PROCESSING → COMPLETED/FAILED → REFUNDED
  • Single transition() method — every status change validated, persisted, and audited in one place
  • Atomic operations: transaction + audit event + outbox event in one DB commit

Reliability

  • Idempotency keys prevent duplicate charges (application check + DB UNIQUE constraint)
  • Circuit breaker on bank API (CLOSED → OPEN → HALF_OPEN) prevents cascading failures
  • Retry with exponential backoff + jitter avoids thundering herd on recovery
  • Transactional outbox pattern with Kafka publishing support

Security & Auth

  • API key authentication per merchant (Bearer token → merchant lookup → context)
  • Token bucket rate limiting per IP
  • Non-root Docker container

Observability

  • Structured JSON logging (slog) with request ID correlation
  • Prometheus metrics at GET /metrics — http_requests_total, http_request_duration_seconds (histogram with p50/p95/p99), http_active_requests, plus Go runtime metrics (GC, goroutines, memory); all scraped by a single promhttp.Handler()
  • OpenTelemetry distributed tracing — OTLP HTTP export to any compatible backend (Jaeger, Grafana Tempo); enabled via environment variables
  • Health check endpoint for K8s liveness/readiness probes

Rate Limiting

PayFlow uses a token bucket rate limiter backed by Redis and Lua so the decision is atomic and shared across instances. Each client key is evaluated against the same distributed state, which keeps enforcement consistent even when the API runs with multiple replicas.

The limiter is intentionally fail-open at the application boundary: if Redis is unavailable, the request is allowed to proceed rather than dropping traffic due to a control-plane dependency.

Idempotency

Idempotency prevents duplicate execution of payment requests. Redis acts as the coordination layer with in_progress and completed states, while PostgreSQL remains the source of truth for the transaction itself.

The in_progress state blocks concurrent duplicate work, and the completed state returns the stored response for repeated requests. TTL-based storage allows safe recovery if the process or Redis node restarts.

Architecture Notes

Lua keeps the Redis operations atomic, which matters for both token bucket updates and idempotency state transitions. Redis provides distributed consistency for rate limiting and request coordination without moving business correctness out of the database.

Rate limiting fails open to preserve availability. Idempotency uses TTLs and safe recovery semantics so transient Redis loss does not corrupt payment correctness, and PostgreSQL remains the durable record of the final result.

Tech Stack

Go 1.26.5 · chi · PostgreSQL 16 · Redis 7 · Kafka · Docker · Kubernetes · Prometheus · Grafana · Jaeger · OpenTelemetry

Quick Start

git clone https://github.com/aszender/payflow.git
cd payflow
docker-compose up -d --build

The full stack starts automatically:

Service URL Credentials
PayFlow API http://localhost:8080 Bearer token (see below)
Grafana http://localhost:3000 admin / admin
Prometheus http://localhost:9090 —
Jaeger UI http://localhost:16686 — 
# Send a payment (generates metrics and traces)
curl -X POST http://localhost:8080/api/v1/transactions \
  -H "Authorization: Bearer sk_live_maple_001" \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: order_12345" \
  -d '{"amount_cents":15000,"currency":"CAD"}'

# Open Grafana → PayFlow dashboard → see request rate, p99 latency, error rate live
# Open Jaeger → search service "payflow" → see the full trace for that request

Kubernetes

The repository includes a runnable Kubernetes stack under deployments/k8s for a local or single-node cluster. It deploys the application plus PostgreSQL, Redis, Kafka in KRaft mode, persistent volumes for PostgreSQL and Kafka, and a bootstrap Job that creates the payment-events topic.

High-level flow:

# Build the application image
docker build -t payflow:latest .

# If your cluster does not share the host Docker daemon, load or push the image
# Example for kind:
kind load docker-image payflow:latest --name <your-cluster-name>

# Apply the stack
kubectl apply -k deployments/k8s

# Wait for workloads
kubectl -n payflow get pods

# Access the API locally
kubectl -n payflow port-forward svc/payflow 8080:80

See deployments/k8s/README.md for the full deployment notes.

API

Method Endpoint Auth Description
GET /health No Health status with database check
GET /ready No Readiness probe
GET /metrics No Prometheus metrics endpoint
POST /api/v1/transactions Bearer Create payment
GET /api/v1/transactions/{id} Bearer Get transaction
POST /api/v1/transactions/{id}/refund Bearer Refund (atomic reverse)
GET /api/v1/transactions/{id}/events Bearer Audit trail
GET /api/v1/merchants/{id}/balance Bearer Merchant balance
GET /api/v1/merchants/{id}/transactions Bearer Paginated list

Example: Create a payment

The authenticated merchant comes from the Bearer API key, so the request body does not include merchant_id.

curl -X POST http://localhost:8080/api/v1/transactions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_live_maple_001" \
  -H "X-Idempotency-Key: order_12345" \
  -d '{
    "amount_cents": 15000,
    "currency": "CAD"
  }'
{
  "success": true,
  "data": {
    "id": "tx_a1b2c3d4",
    "merchant_id": "m_001",
    "amount_cents": 15000,
    "currency": "CAD",
    "status": "COMPLETED",
    "idempotency_key": "order_12345",
    "created_at": "2025-03-01T10:30:00Z"
  }
}

API Contract

The HTTP contract is documented in api/openapi.yaml.

Use that file as the source of truth for endpoint paths, request bodies, and response shapes.

Environment Variables

The repository includes a root-level .env.example file with safe placeholder values for local development. It documents the expected environment variables used by the application so you can create a private .env file without guessing the required keys.

The example file includes:

  • DB_HOST
  • DB_PORT
  • DB_USER
  • DB_PASSWORD
  • DB_NAME
  • PORT

Observability Stack

docker-compose up starts the full observability stack alongside the application.

Grafana — http://localhost:3000 (admin / admin)

Pre-built PayFlow dashboard (observability/grafana/dashboards/payflow.json) loads automatically:

  • Top row: request rate, error rate, p99 latency, active requests — all live with color thresholds (green/yellow/red)
  • Request rate by endpoint — time series broken down by method + path
  • Latency percentiles — p50 / p95 / p99 on one graph
  • HTTP responses by status code — 2xx green, 4xx yellow, 5xx red
  • p99 latency per endpoint — spot which route is slow
  • Go runtime: goroutines, heap memory, GC pause p99

Prometheus — http://localhost:9090

Scrapes app:8080/metrics every 15 seconds. Application metrics:

  • http_requests_total{method, path, status} — counter
  • http_request_duration_seconds{method, path} — histogram (fixed buckets)
  • http_active_requests — gauge

Useful PromQL queries:

# p99 latency for POST /api/v1/transactions
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{path="/api/v1/transactions"}[5m]))

# error rate
sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

# requests per second by endpoint
sum by (method, path) (rate(http_requests_total[1m]))

Jaeger — http://localhost:16686

Receives OTel traces from PayFlow over OTLP HTTP. Search by service payflow to see the full request waterfall: middleware → service → database. Tracing is enabled automatically in docker-compose via:

TRACING_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4318

When running without docker-compose, tracing is off by default (TRACING_ENABLED unset) — zero overhead.

Readiness

The API exposes a GET /ready endpoint that returns:

{
  "status": "ready"
}

It responds with HTTP 200 and is intended for readiness checks.

Project Structure

payflow/
├── .github/
│   └── workflows/
│       └── ci.yml                    ← CI pipeline
├── cmd/server/main.go                ← wires everything, graceful shutdown
├── internal/
│   ├── domain/
│   │   ├── models.go                 ← Transaction, Merchant, state machine
│   │   └── errors.go                 ← sentinel errors (ErrInvalidAmount, etc.)
│   ├── repository/
│   │   ├── interfaces.go             ← DBTX + repository contracts
│   │   ├── postgres/
│   │   │   ├── db.go                 ← connection pool + health check
│   │   │   ├── merchant.go           ← merchant queries
│   │   │   ├── transaction.go        ← tx queries + pagination
│   │   │   ├── events.go             ← audit event and outbox queries
│   │   │   └── postgres_integration_test.go
│   │   └── mock/
│   │       └── repos.go              ← in-memory implementations for testing
│   ├── service/
│   │   ├── payment.go                ← business logic, state machine
│   │   ├── bank.go                   ← bank client implementations
│   │   ├── payment_test.go           ← service tests
│   │   ├── resilience.go             ← circuit breaker, retry
│   │   ├── resilience_test.go        ← resilience tests
│   │   └── outbox_worker.go          ← Kafka publisher
│   ├── handler/
│   │   ├── handlers.go               ← HTTP handlers, error mapping
│   │   └── handler_test.go           ← endpoint tests with httptest
│   ├── middleware/
│   │   ├── middleware.go             ← auth, logging, rate limit, recovery, CORS
│   │   ├── redis_rate_limiter.go     ← Redis token bucket limiter
│   │   ├── idempotency_store.go      ← Redis idempotency coordination
│   │   ├── idempotency_middleware.go ← cached replay + in-progress protection
│   │   ├── rate_limit.lua            ← atomic token bucket script
│   │   └── idempotency_check.lua     ← atomic idempotency check script
│   ├── metrics/
│   │   └── metrics.go                ← counters, histograms, gauges
│   ├── telemetry/
│   │   └── tracing.go                ← OpenTelemetry setup, OTLP HTTP export
│   ├── config/
│   │   └── config.go                 ← env-based configuration
│   └── concurrency/
│       ├── patterns.go               ← worker pool, verification pipeline
│       └── patterns_test.go          ← concurrency tests with -race
├── observability/
│   ├── prometheus.yml                ← scrape config targeting app:8080/metrics
│   └── grafana/
│       ├── provisioning/             ← auto-loads datasource + dashboard on startup
│       └── dashboards/
│           └── payflow.json          ← pre-built dashboard (latency, errors, runtime)
├── migrations/                       ← 6 manual SQL migration files (idempotent, ordered)
├── Dockerfile                        ← multi-stage container build
├── docker-compose.yml                ← PostgreSQL + Kafka + Redis + Prometheus + Grafana + Jaeger
├── Makefile
└── go.mod

Key Patterns Implemented

DBTX Interface

Repositories accept an interface satisfied by both *sql.DB and *sql.Tx. The service layer calls repo.WithTx(tx) to run multiple repositories inside one database transaction. Creating a payment atomically writes the transaction record, audit event, and outbox event — if any fails, all roll back.

Transactional Outbox

Events are written to an outbox table in the same DB transaction as the payment. A background worker polls unpublished events and publishes to Kafka. This demonstrates the transactional outbox pattern and decouples event delivery from the request path.

State Machine

Every status transition goes through a single transition() function that: validates the transition is legal, updates the database, updates the in-memory object, and records an audit event. Impossible to skip a state or make an invalid transition.

Circuit Breaker

The bank client is wrapped in a circuit breaker. After N consecutive failures, the circuit opens and subsequent calls fail immediately instead of waiting for timeout. After a cooldown period, a test request is allowed through — if it succeeds, the circuit closes.

Design Decisions and Trade-offs

Decision Chose Over Why
Database PostgreSQL MongoDB ACID transactions required for financial data. Money needs consistency, not eventual consistency
ORM database/sql GORM Full control over queries and transactions. ORMs hide SQL complexity that matters in payment systems
Framework chi gin/echo Lightweight, stdlib-compatible. net/http signatures, no vendor lock-in
Architecture Hexagonal Layered MVC Repository interfaces enable testing without DB. Swap PostgreSQL for another persistence layer without touching business logic
Bank call Sync + outbox Fully async Client gets immediate response. Outbox preserves downstream delivery intent even if Kafka is temporarily unavailable
Testing Mocks + integration tests Testcontainers-only Unit tests stay fast, and PostgreSQL integration tests run in CI
Rate limiting Redis token bucket + Lua In-memory token bucket Shared atomic limits across instances without changing business logic

Testing

go test -race -cover ./...
go test -v ./internal/service/
go test -v ./internal/handler/

Most tests run without Docker or PostgreSQL. PostgreSQL integration tests run when TEST_DATABASE_URL is set and are also wired into CI. The test suite covers service flows, validation errors, idempotency, refunds, balance tracking, state transitions, circuit breaker behavior, handler behavior, and repository integration paths.

CI

GitHub Actions runs the automated validation pipeline on every push and pull request:

  • go test ./...
  • go test -race ./...
  • PostgreSQL-backed integration tests
  • Docker image build verification

Production Considerations

Things this project demonstrates vs. what a production system would add:

This Project Production
Simulated bank client by default Real bank partner API integration
Redis-backed global rate limiter More granular route- and merchant-level policies
JSON logging, Prometheus metrics, preconfigured Grafana dashboard, and OpenTelemetry tracing Add Alertmanager rules and Datadog/CloudWatch export
Single PostgreSQL Primary + read replicas + PgBouncer
Basic API key auth OAuth2 + mTLS + PCI DSS compliance
Manual SQL migrations executed at startup golang-migrate or Atlas
Basic K8s manifests included in repo Fully hardened deployment specs and autoscaling

License

MIT

About

Production-style payment gateway API in Go — idempotent processing, transactional outbox, circuit breaker, Kafka, Redis, Prometheus/OpenTelemetry observability, and full audit trail.

Topics

circuit-breakerdistributed-systemsfintechgohexagonal-architecturekafkamicroservicesopentelemetrypayment-gatewayredis

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages