Skip to content

Persistent indexer storage (PostgreSQL backend) #31

Description

@LiranCohen

Summary

Replace the in-memory `IndexerStore` with a persistent storage backend (PostgreSQL) for production deployments.

Context

The current `IndexerStore` (`src/indexer/store.ts`) uses in-memory `Map`/`Set` collections with FIFO eviction (added in the hardening PR #22). This is fine for development but loses all indexed data on restart. A production indexer needs persistent storage.

Scope

  • Create `IndexerStorePostgres` implementing the same public API as `IndexerStore`
  • Use the existing PostgreSQL infrastructure from `docker-compose.test.yaml`
  • Add database migrations for the indexer schema (repos, stars, follows, cursors)
  • Make the store backend configurable via environment variable (`DWN_GIT_INDEXER_STORE=postgres`)
  • Keep in-memory store as default for development

Design considerations

  • The store interface is already well-defined — `putRepo`, `putStar`, `putFollow`, `getRepo`, `search`, etc.
  • Full-text search (`search()` method) maps naturally to PostgreSQL `tsvector`/`tsquery`
  • Trending calculation can use SQL window functions
  • Connection pooling for the crawl loop + API server

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions