start.md

Project 1 — Notes CRUD REST API

Roadmap for this Project

• Conventions: Standard Go project layout, semantic versioning, conventional commits, golangci-lint, unit + integration tests, docker-compose for deps, .env for secrets (dev only).

• Focus: HTTP fundamentals, persistence, caching, background jobs, basic auth, logging, metrics, tests.

• Cross-cutting tech: net/http or Gin/Fiber, PostgreSQL (pgx/GORM), SQLite for quick start, Redis, MinIO (S3), Swagger/OpenAPI, Wire (optional DI), testify, httptest, golang-migrate, air or reflex for live reload.

About the Project

Notes CRUD REST API

• Summary: A simple notes service exposing RESTful CRUD with pagination and filtering.
• Goals: Routing, handlers, repo layer, migrations, DTO vs model, input validation, errors.
• Stack: Gin (or chi), PostgreSQL + pgx, golang-migrate, zap/zerolog, OpenAPI.
• Milestones:
  1. Bootstrap server + healthz
  2. CRUD endpoints + validation (go-playground/validator)
  3. DB migrations + repository layer (context, timeouts)
  4. Structured logs + error handling + 12-factor config
  5. OpenAPI spec + swagger UI
  6. Unit + integration tests (spin up Postgres via docker compose)
• Stretch: ETag/If-Modified-Since; cursor pagination; graceful shutdown hooks.
• Deliverables: API + README with run commands, OpenAPI JSON/YAML, test report.

Project Setup

This section contains the complete plan and system design for the Notes CRUD REST API. It outlines requirements, API design, data model, architecture, tooling, project structure, and a step-by-step execution plan with acceptance criteria. No code is included here.

---

Objectives & Scope

• Build a clean, maintainable REST API for notes with pagination, filtering, and sorting.
• Prioritize correctness, observability, and good Go practices (layering, context, validation, migrations).
• Deliver OpenAPI docs, unit + integration tests, and Docker Compose for local dev.
• Out of scope initially: auth, multi-tenancy, full-text search (considered as stretch goals).

---

Functional Requirements

• Create, read, update, delete a note.
• List notes with pagination (page/page_size), optional text search (title/content), tag filtering, and sorting by created_at/updated_at asc/desc.
• Health endpoint for readiness/liveness checks.
• Consistent error payloads and HTTP status code mapping.

Non-Functional Requirements

• Performance: Local P50 < 20ms, P95 < 100ms for CRUD paths with Postgres.
• Reliability: Graceful shutdown; context deadlines for DB calls; idempotent delete.
• Observability: Structured logs, Prometheus metrics, and basic traces.
• Security: Input validation, size limits, parameterized SQL, conservative CORS in dev only.
• Maintainability: Layered architecture, small interfaces, table-driven tests, linting.

---

API Design (v1)

Base URL: /v1

Note fields:

• id (uuid), title (1..200), content (0..10000), tags (array[string] ≤ 10), is_archived (bool), created_at, updated_at (RFC3339)

Endpoints:

• POST /v1/notes → 201 Created; body: { title, content, tags?, is_archived? }
• GET /v1/notes/{id} → 200 OK or 404
• PATCH /v1/notes/{id} → partial update; 200 OK or 400/404
• DELETE /v1/notes/{id} → 204 No Content (idempotent)
• GET /v1/notes → list with page, page_size(≤100), q, tags, sort(created_at|updated_at), order(asc|desc)
  • Response: { data: Note[], pagination: { page, page_size, total, next_page? } }
• GET /healthz → { status: "ok" }

Error shape: { "error": { "code": "VALIDATION_ERROR|NOT_FOUND|INTERNAL", "message": "...", "details": { } } }

---

Data Model (PostgreSQL)

Table: notes

• id UUID PRIMARY KEY
• title TEXT NOT NULL CHECK (length(title) BETWEEN 1 AND 200)
• content TEXT NOT NULL CHECK (length(content) <= 10000)
• tags TEXT[] NOT NULL DEFAULT '{}'
• is_archived BOOLEAN NOT NULL DEFAULT FALSE
• created_at TIMESTAMPTZ NOT NULL DEFAULT now()
• updated_at TIMESTAMPTZ NOT NULL DEFAULT now()

Indexes:

• notes_updated_at_idx on (updated_at DESC)
• Optional: notes_tags_gin_idx GIN on tags (if tag filtering is used frequently)
• Stretch: tsvector for full-text search

Pagination:

• Start with page/size. Add keyset/cursor pagination as a stretch goal if needed.

---

Architecture

Layers:

• HTTP (Gin): routing, middleware (request ID, recovery, logging), binding/validation, response shaping.
• Service: business rules, input sanitation, pagination defaults, orchestration.
• Repository: Postgres queries via pgx, transactions, context timeouts, error mapping.
• Infra: config, logger, metrics/tracing setup, DB pool, migrations.

Context & shutdown:

• Per-request context propagated to DB; timeouts on handlers and queries; graceful shutdown waits for in-flight requests.

Observability:

• Logs: structured (zerolog) with request_id, path, status, latency.
• Metrics: HTTP request counter and latency histogram; DB query duration histogram; build info.
• Tracing: minimal spans for request and DB operations via OpenTelemetry.

Error handling:

• Central mapping from domain/validation errors to HTTP status and standard error envelope.

---

Technology Choices and Why

• PostgreSQL vs MySQL

  • Postgres offers arrays (for tags), JSONB, rich indexing (GIN), robust transactional semantics, and native full-text (tsvector) for future stretch goals. These features simplify filters and future search more than MySQL equivalents.

• pgx vs GORM

  • pgx gives precise SQL control, predictable performance, and teaches core SQL patterns. For a beginner backend, explicit queries improve understanding and reduce ORM magic. GORM could be added later if speed of development becomes a priority.

• Gin vs chi/Fiber

  • Gin is fast, popular, and has built-in binding/validation helpers and a rich middleware ecosystem. chi is a strong alternative; we choose Gin for wider examples/tutorials.

• Migrations: golang-migrate

  • Mature, up/down SQL, CI-friendly, supports many drivers.

• Logging: zerolog (zap also fine)

  • Structured JSON logs with low overhead; easy to add fields for context.

• Validation: go-playground/validator

  • De facto standard with tag-driven rules; integrates with Gin binding.

• API Docs: OpenAPI (swaggo or hand-authored spec)

  • Ensures contract clarity; enables Swagger UI for manual testing in dev.

• Observability: OpenTelemetry + Prometheus

  • Standard traces/metrics; easy local scraping via /metrics.

---

Project Structure (planned)

notes-api/

• cmd/
  • notes-api/ (main entrypoint)
• internal/
  • config/ (env loading, defaults, validation)
  • http/ (router setup, middleware, handler wiring)
  • notes/
    • handler.go (HTTP handlers)
    • service.go (business logic)
    • repository.go (pgx queries)
    • models.go (domain models, DTOs)
  • db/
    • migrations/ (SQL up/down files)
    • pool.go (pgxpool init)
  • observability/
    • logging.go
    • metrics.go
    • tracing.go
  • version/
    • version.go
• deployments/
  • docker-compose.yml (Postgres, app, optional Prometheus)
• docs/
  • openapi.yaml -.env.example
• Makefile
• README.md

Configuration (12-factor):

• HTTP_ADDR (e.g., :8080)
• DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME, DB_SSLMODE
• DB_MAX_OPEN_CONNS, DB_MAX_IDLE_CONNS, DB_CONN_MAX_LIFETIME
• LOG_LEVEL (debug|info|warn|error)
• METRICS_ENABLED (bool), OTEL_EXPORTER_OTLP_ENDPOINT (optional)
• PAGINATION_DEFAULT_LIMIT, PAGINATION_MAX_LIMIT

Docker Compose (dev):

• Services: postgres (with volume, healthcheck), notes-api (depends_on), optional prometheus
• Ports: 5432 (db), 8080 (api)

---

Step-by-Step Execution Plan (no code)

0. Repo bootstrap & tooling

• Init go module; add Makefile targets: lint, test, run, compose-up/down.
• Add .env.example; set up golangci-lint config; set VSCode tasks (optional).
• Acceptance: make lint and make test run without errors.

1. Config, logging, health

• Implement config loader with defaults; setup zerolog; request ID, recovery middleware; /healthz endpoint.
• Graceful shutdown on SIGINT with context timeout.
• Acceptance: GET /healthz returns {status:"ok"}; shutdown logs orderly.

2. DB & migrations

• Write SQL for notes table and indexes; integrate golang-migrate (CLI or via code) for dev.
• Acceptance: migration applies cleanly; re-running is idempotent.

3. Repository layer (pgx)

• Setup pgxpool; CRUD methods with context timeouts; map SQL errors to domain errors.
• Acceptance: repository unit tests pass against a test DB; timeouts enforced.

4. Service layer

• Validate inputs (lengths, tags count); apply defaults; orchestrate repository; define domain errors.
• Acceptance: table-driven unit tests pass (valid/invalid paths).

5. HTTP layer

• Routes for CRUD and list; binding/validation errors map to 400; not found → 404; success → 2xx with JSON envelope.
• Acceptance: handler tests (httptest) cover success and error paths.

6. Pagination/filter/sort

• Enforce page_size bounds; support q and tags; add necessary indexes.
• Acceptance: integration tests confirm correct filtering and pagination metadata.

7. OpenAPI docs

• Generate or author openapi.yaml; serve Swagger UI in dev; ensure examples & schemas match responses.
• Acceptance: spec validates and matches behavior.

8. Observability

• Expose /metrics; add HTTP request and DB histograms; minimal traces.
• Acceptance: metrics visible; logs structured with request_id; traces export locally.

9. Integration tests & CI

• Use docker-compose Postgres; run migrations; run API in test mode; black-box HTTP tests.
• Acceptance: CI pipeline runs lint, unit, integration tests successfully.

10. Containerization & local run

• Multi-stage Dockerfile; compose service for app + db; healthchecks.
• Acceptance: docker compose up exposes API; smoke tests pass.

11. Performance sanity

• Quick load test (k6/Vegeta) for CRUD/list; profiling notes captured.
• Acceptance: meets NFRs; findings documented in README.

12. Stretch goals

• ETag/If-None-Match for GETs; cursor pagination; improved shutdown hooks; request logging redaction.
• Acceptance: tests updated; docs reflect changes.

---

Testing Strategy

• Unit: service and repository with table-driven tests; mocks for repo when testing service.
• Integration: spin Postgres via compose, apply migrations, black-box API tests.
• Negative tests: invalid payloads, oversized fields, bad query params, missing IDs.
• Coverage goal: ≥ 70% across service/repo/handlers.

---

Observability Plan

• Logging fields: request_id, method, path, status, latency_ms, db_time_ms.
• Metrics: http_requests_total, http_request_duration_seconds (histogram), db_query_duration_seconds (histogram), build_info.
• Traces: basic spans for request → service → repo; propagate W3C trace context.

---

Security & Compliance

• Validation: strict lengths and tag counts; reject unknown JSON fields; Content-Length limits.
• SQL safety: parameterized queries; least-privileged DB user.
• CORS: disabled unless explicitly required; allow localhost in dev only.
• Secrets: never commit; use .env for dev; document in .env.example.

---

Risks & Mitigations

• Migration drift → run migrations in CI and dev startup; keep SQL reviewed.
• Pagination scalability → add indexes; switch to keyset pagination if needed.
• Validation gaps → comprehensive table-driven tests and OpenAPI schema alignment.

---

Deliverables

• OpenAPI spec, README with run/test instructions and architecture summary.
• Docker Compose for Postgres + service; Makefile targets.
• Unit + integration tests with CI workflow outline.

---

Glossary

• DTO: input/output binding structures.
• Repository: data-access layer with explicit SQL.
• Service: business logic orchestrating validation and persistence.