Skip to content

fix: process repositories in batches to prevent partial sync on large orgs#1029

Open
roryharness wants to merge 2 commits into
github-community-projects:main-enterprisefrom
roryharness:fix/batch-repo-processing-to-prevent-partial-sync
Open

fix: process repositories in batches to prevent partial sync on large orgs#1029
roryharness wants to merge 2 commits into
github-community-projects:main-enterprisefrom
roryharness:fix/batch-repo-processing-to-prevent-partial-sync

Conversation

@roryharness

Copy link
Copy Markdown

Problem

SettingsRepository.eachRepositoryRepos fetches every repository in the installation and processes them all at once with a single Promise.all(...) over the full list. On large organizations this has two problems:

  1. Unbounded concurrency — thousands of checkAndProcessRepo calls fire simultaneously, creating heavy API/memory pressure.
  2. Partial sync on any failurePromise.all rejects as soon as one repository fails, aborting the entire run. Repositories that hadn't been processed yet are silently skipped, leaving the org partially synced.

Fix

  • Process repositories in fixed-size batches (CONCURRENCY = 10) instead of all at once, bounding concurrency and API pressure.
  • Use Promise.allSettled per batch so a single repository's failure no longer aborts the whole sync. Failed repositories are logged via log.error and processing continues.
  • Aggregate and return the successfully processed results.

This makes full-org syncs resilient: one bad repository no longer prevents the rest of the org from being synced.

Additional change

  • Add COPY full-sync.js /opt/safe-settings/ to the Dockerfile so the full-sync entrypoint is present in the built image (it was previously omitted).

Behavior / compatibility

  • Behavior for small orgs is unchanged aside from batching; all repositories are still processed.
  • No public API or configuration changes.

… orgs

When an org has many repositories, `eachRepositoryRepos` fires all of
them concurrently via `Promise.all`. This exhausts GitHub's API rate
limit (5,000 req/hr for app installations) mid-sync, causing the
remaining repositories to be silently skipped — leaving them out of
sync with the desired config.

This change processes repositories in chunks of 10 using
`Promise.allSettled`, so:

1. Concurrent API pressure stays well within rate limits
2. A single repo failure no longer aborts the entire sync — errors are
   logged individually and processing continues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant