A transient failure in the final `git push` discards a completed upload

[singlesp]

What went wrong?

Summary

push() retries annex object transfers, but the two git.push() calls at the end of the same function have no retry and no error handling. If either fails once the whole run aborts, and there is no way to resume without repeating it.

This happened on a 226 GB / 65,558-file upload. All 40,176 annex objects transferred successfully over ~4.5 hours (using the patched code from #4048 ), the git-annex branch pushed, and then the main push failed:

Transferring annexed files 100.00%  16277.0s 40176/40176
INFO Stored 11063166 bytes for key "SHA256E-s11063166--97927a2d..." from path "..."
INFO Fetching git-annex branch updates.
INFO Merging git-annex branch.
INFO Pushing git-annex branch...
Pushing changes...
error: Uncaught (in worker "") (in promise) Error: Request timed out
    at ClientRequest.<anonymous> (simple-get/4.0.1/index.js:76:8)
    at TLSSocket.emitRequestTimeout (node:_http_client:1152:9)
    at TLSSocket.Socket._onTimeout (ext:deno_node/net.ts:1259:10)
error: Uncaught (in promise) null

The remote was left with refs/heads/git-annex pushed and every annex object stored, but refs/heads/main still pointing at the initial [OpenNeuro] Dataset created commit — a dataset that looks empty despite 226 GB of content sitting on the remote.

Why the retry is worth having

The same function already retries the per-key transfers three times:

      let storeKeyResult = -1
      let retries = 3
      while (storeKeyResult === -1 && retries > 0) {
        retries -= 1
        storeKeyResult = await storeKey(...)
        if (storeKeyResult === -1 && retries > 0) {
          logger.warn(`Failed to transfer annex object "${key}" - retrying`)
        }
      }

but the pushes do not:

    logger.info("Pushing git-annex branch...")
    await git.push({ ...context.config(), ref: "git-annex", onMessage: console.log })
...
    console.log("Pushing changes...")
    await git.push({ ...context.config(), onMessage: console.log })

The timeout here is not something the CLI sets: isomorphic-git's node client passes no timeout option ({url, method, headers, agent, body}), so the limit comes from Deno's node:http compatibility layer, which imposes a default Node itself does not have. That default is not configurable from the CLI, and the larger the tree, the longer the server takes to respond — so bigger datasets are more likely to trip it.

Recovery is possible but undocumented

The data was recoverable only because the working repository still existed. Pushing the same ref with system git succeeded immediately:

$ git --git-dir=<preserved>/ds008547.git \
      -c credential.useHttpPath=true -c credential.helper='!openneuro git-credential' \
      push https://openneuro.org/git/2/ds008547 main:main
To https://openneuro.org/git/2/ds008547
   862736dd2..2b25bfebd  main -> main

System git has no such timeout, so the same packfile went through without trouble. But the CLI's working repo lives in a Deno.makeTempDir() directory, which on a cluster is node-local and wiped when the job ends — so by default there is nothing left to retry from, and the only option is to redo the entire upload.

Suggestions

1. Retry the two git.push() calls, mirroring the existing storeKey retry.
2. Report the failure clearly rather than as Uncaught (in promise) null, and say which ref failed and that annex content is already stored.
3. Optionally, on a failed push, print the repo path and the git push command needed to finish manually, since the temp directory is otherwise not obvious.

Happy to open a PR for any of the these if that would be useful.

Environment

• OpenNeuro CLI 5.4.0, Deno 2.9.0, Linux x86_64
• 65,558 files / 226 GB, 40,176 annex keys

Expected behavior

n/a

How to reproduce

No response

Desktop

Phone

Additional information

No response