Skip to content

Refresh cadence & prioritization policy (background updates) #107

Description

@peetzweg

Part of #106 — the policy half of the background-refresh system: who gets refreshed and how often.

Problem

Today refresh is lazy-on-lookup only. Entities nobody visits go stale (rank, followers, commit tail); org totals never refresh on the request path. We need a proactive, prioritized cadence that keeps the visible surface fresh while staying polite to GitHub's ~5,000 points/hr budget (shared token).

Proposal (to ratify / adjust)

Tiers, highest priority first:

  1. Hot — entities looked up in the last N days (from lookups.searchedAt) + the top ~250 of each leaderboard. Refresh weekly.
  2. Long tail — everything else. Refresh monthly (stalest lastFetched/builtAt first).
  3. Profile metadata (followers, rank inputs, bio) drifts faster than the monthly contribution curve — consider a more frequent, cheaper pass (one profile fetch/entity) independent of the full contribution re-fetch.

Budget: reserve a floor for live traffic (our scripts already use REMAINING_FLOOR = 500); split the remaining hourly budget across tiers, hot first.

Applies to users AND orgs — org contribution totals must join the cadence (they currently never refresh on the request path).

Acceptance criteria

  • Written cadence policy: which tiers, which interval each, how the budget is split
  • Prioritization query defined (recent lookups + top-N leaderboard + stalest-first long tail), for both kind='user' and kind='org'
  • Idempotent + checkpointed so a killed run resumes (mirror backfill-contributions.ts / backfill-orgs.ts hygiene)
  • Decide the in-progress-month semantics interaction (Show the current in-progress month as a dashed 'trend' segment #35)

Relationship to existing issues

Generalizes #6 (monthly refresh of cached users) into a tiered, org-aware policy; feeds #58 (retroactive reconcile) which is the "re-fetch immutable past months" special case; the runner is sub-issue B.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions