Skip to content

perf(cli): Batch index adds instead of staging one file at a time - #4056

Merged
nellh merged 1 commit into
OpenNeuroOrg:masterfrom
singlesp:feat/batch-adds-only
Aug 6, 2026
Merged

perf(cli): Batch index adds instead of staging one file at a time#4056
nellh merged 1 commit into
OpenNeuroOrg:masterfrom
singlesp:feat/batch-adds-only

Conversation

@singlesp

@singlesp singlesp commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #4055

Problem

annexAdd stages each file with its own git.add, and isomorphic-git
re-serializes and rewrites the entire index on every call. Staging is therefore
O(n) per file and O(n²) overall.

On a 185k-file dataset the rate fell from ~300 files/min to ~30 files/min by 83k
files, projecting roughly four more days of staging before any transfer started.

Change

Collect paths and stage them in batches of 500. annexAdd no longer calls
git.add itself; it returns true and the caller batches. git.add already
accepts an array, so no upstream change is required.

Measurement

isomorphic-git 1.36.3, 20,000 files, local NVMe:

per-file add    1173.3s    rate decays 8840 -> 541 files/min   (rate x n constant)
batched (1000)     6.1s    rate flat, ~180k files/min

192x faster, and the rate no longer decays. Confirmed in production on a 185k-file
dataset: ~200 files/min and falling became ~1200 files/min and steady.

Testing

  • deno test — 36 passed, 0 failed
  • deno fmt --check and deno lint clean on the changed files
  • Ran the end-to-end worker test with ADD_BATCH_SIZE set to 1, 2 and 500 to
    exercise both the multi-flush and single-flush paths. All produce an identical
    repository — that test asserts an exact git object count, so a staging error
    would surface.

Note for review

With batching, up to ADD_BATCH_SIZE - 1 files can be staged on disk but not yet
recorded in the index if the process dies mid-run. Staging is idempotent so a rerun
redoes them. 500 is a compromise between that window and the per-flush cost; the
benchmark is already flat by ~2000 files, so a smaller batch would cost nothing
measurable if you would prefer a tighter bound.

isomorphic-git re-serializes and rewrites the entire index on every
git.add, so staging one path per call costs O(n) per file and O(n^2)
overall. On large datasets this dominates the upload: the staging rate
decays as 1/n, from ~300 files/min early on to ~30 files/min at 83k
files.

Collect paths and stage them in batches of 500 instead. annexAdd no
longer stages the file itself, leaving that to the caller.

Measured with isomorphic-git 1.36.3 on local NVMe, 20k files:

  per-file add   1173.3s   rate decays 8840 -> 541 files/min
  batched (1000)    6.1s   rate flat, ~180k files/min

git.add already accepts an array of paths, so this needs no upstream
change.
@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.75000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.89%. Comparing base (28eb000) to head (34d5229).

Files with missing lines Patch % Lines
cli/src/worker/git.ts 68.75% 2 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4056      +/-   ##
==========================================
- Coverage   48.89%   48.89%   -0.01%     
==========================================
  Files         686      686              
  Lines       38069    38075       +6     
  Branches     1894     1897       +3     
==========================================
+ Hits        18615    18616       +1     
- Misses      19293    19295       +2     
- Partials      161      164       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nellh
nellh merged commit 42bdc94 into OpenNeuroOrg:master Aug 6, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Staging is quadratic in file count: git.add is called once per file

2 participants