New version of bscp that adds a lot of functionality (note: AI assisted)#22
Open
groenewe wants to merge 62 commits into
Open
New version of bscp that adds a lot of functionality (note: AI assisted)#22groenewe wants to merge 62 commits into
groenewe wants to merge 62 commits into
Conversation
Complete rewrite replacing the original single-pass push-only script: - Both push (local→remote) and pull (remote→local) modes, auto-detected from which argument carries the HOST: prefix. - Protocol header (HEADER_FMT '<QQQQQQB', 49 bytes) carries size, blocksize, section_size, start_offset, filename_len, hashname_len, and a mode byte; replaces the old 4×uint64 header. - _remote() is a real Python function extracted at runtime with inspect.getsource(); the remote shell tries python3/python2/python in order. Compatible with Python 2.6+ and 3.x. - Section-based processing: file processed in --section-size chunks (default 10 GiB) so that peak memory scales with diff density per section, not the whole file. Push caps section size to half available RAM automatically. - Windowed pull (PULL_WINDOW=128) to avoid deadlock on the SSH pipe. - Progress reporting: scan phase shows MiB/s + ETA; copy phase shows KiB/s. - Resume on failure: ConnectionLost carries the section start offset and prints a copy-pasteable --resume-from command. - Auto-retry with exponential back-off (--retries N). - Dry-run (-N): count differing blocks without writing. - Block-count limit (-B N): stop after N blocks (useful for partial syncs). - argparse CLI: -b, -s, -a, -r, --retries, -i, -o, -C, -N, -B, -p. Also adds: - .github/README.md: full usage reference with option table and examples. - PROTOCOL.md: wire-format specification and design rationale. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Allow the destination to be smaller than the source. Without this flag, bscp rejects a push when remote_size < local_size. With it: - Push and pull both use sync_size = min(local_size, remote_size); content beyond the smaller side is silently skipped. - A warning is printed when the destination is the limiting factor. - The ALLOW_SMALLER flag (bit 1 of the mode byte) is set in the protocol header so the remote side applies the same sync_size logic. - --allow-smaller is preserved in the auto-generated resume command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
With --reread, the push path stores only the differing block offsets (8 bytes each) during Phase A instead of buffering the full block data. During Phase B it re-reads each differing block from disk before sending. Trade-off: a second sequential disk read in exchange for O(diff_count × 8) peak memory instead of O(diff_count × blocksize). Useful when the source file is large and RAM is limited. When --reread is active the automatic section-size memory cap is skipped, since the cap was only needed to bound the diff_blocks list size. --reread is preserved in the auto-generated resume command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents the architecture, protocol constants, wire-format invariants, memory model, remote script constraints, and testing procedures. Intended to give Claude (or any future contributor) the context needed to make correct changes without introducing protocol breaks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added module-level quiet = False and batch = False globals - report() returns immediately when batch is set, or skips \r-starting messages when quiet is set - Added -q / --quiet and --batch argparse arguments (with exit-code table in the --batch help text) - In __main__, sets both globals after parse_args(); -q and --batch are included in any generated resume command - Renamed the pull-loop local variable batch → window to avoid shadowing the new module-level flag - Exit codes: no-args usage error → 2; connection lost (non-interrupted) → 3; fatal errors stay 1; SIGINT stays 130 README.md: - Two new rows in the options table (-q / --quiet and --batch) - New "Exit status" section documenting codes 0/1/2/3/130 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t checking for smaller local size
Symmetric size validation — replaced four separate push/pull if branches with a single (src_size, dst_size) check derived from mode. Pulled the cleanup-and-raise pattern into a fail() closure that closes proc.stdin before waiting (the original code could deadlock on proc.wait() after a sanity-digest mismatch because the remote was still blocked on stdin.read(2)). do_sync Phase B — extracted the three duplicated copy-progress blocks into a single show_copy_progress() closure, dropped the if dry_run: break mid-loop pattern in favour of if not dry_run:, and removed an unused enumerate(...). Bug fixes / robustness: - Removed unused import inspect and unused module-level PUSH_PULL_MASK. - format_size(offset, floor=True) in build_resume_cmd() could lose precision (≥9G threshold truncated non-aligned offsets) — switched to floor=False so the exact byte count is preserved through the round-trip via parse_size. - --buffer auto-clamp now also clamps when section_size == 0 (whole-file mode), and the typo'd close-paren in the low-memory note is gone. - start_offset calc no longer divides by zero when the user passes -s 0. - OSError opening the local file is converted to a friendly Error: Cannot open … instead of a Python traceback. - sync_size == 0 (empty source under --allow-truncate) now early-returns cleanly instead of relying on edge-case math in total_sections. Documentation CLAUDE.md — major rewrite: - Architecture: _remote() is a triple-quoted string, not a real function; inspect.getsource() is no longer mentioned. - Constants: ALLOW_SMALLER → ALLOW_TRUNCATE everywhere. - Memory model table: defaults flipped (default = positions only; --buffer is the heavy mode). - Added a Nuitka builds section describing bscp.amd64 (Ubuntu 22.04 amd64) and bscp.arm64 (Ubuntu 24.04 RPi), the Python 3.14.3 toolchain, and the fact that the prebuilt binaries don't need a local Python install. - New "Symmetric size validation" section documenting the refactor. PROTOCOL.md — ALLOW_SMALLER → ALLOW_TRUNCATE, --reread → --buffer (with inverted semantics), clarified that remote_size == 0 overlaps with the "empty file" case (known limitation), and added a design note about the embedded-string approach being frozen-friendly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
format_size(n, floor=True) — replaced the >= 9 × div heuristic with a "largest unit whose count fits in ≤4 digits" rule. Iterates units smallest→largest and stops at the first count ≤ 9999. This: - never loses precision near a unit boundary (e.g. 9G is now shown as 9216M, not 9G) - preserves the "≥ 2 digits preferred" property except at the forced unit-up boundary you described (10000M → 9G, 10240M → 10G) - gracefully extends to T (TiB) without further special-casing parse_size — added the matching T suffix so values that format_size(floor=False) produces still round-trip cleanly. CLI help string updated to K/M/G/T. Final summary line — in, out, and size now go through format_size(floor=True). A 10 MB transfer reads in=5160 out=64K size=10M (...), a 100 MB transfer reads in=50K out=20M size=100M (...). Sub-10K values display as plain byte counts (no suffix), so small ones stay unambiguous. CLAUDE.md updated to document the new rule. Co-Authored-By: Claude Sonnet 4.7 <noreply@anthropic.com>
tests.sh (new, executable) — a 12-test bash harness covering everything we exercised manually plus a couple I noticed were easy to forget: - push, pull, dry-run, resume, --buffer, --allow-truncate (push and pull, both refusal and acceptance), --batch (silent + exit-0), --block-count (correct -r 4M continue hint), bad args (exit 2), missing local file (friendly error), and format_size/parse_size unit tests via inline Python. - Uses mktemp -d for fixtures, cleans up on exit, and works from any cwd because it resolves bscp relative to the script's own directory. - Preflight checks: python3 on PATH and passwordless ssh localhost. Bails with exit 2 if either is missing. - Failure output prints the test name, the captured stderr, and a final list of failed names. Run with ./tests.sh or BSCP=./bscp.amd64 ./tests.sh to test the Nuitka binaries. CLAUDE.md — replaced the "Manual functional tests" walkthrough with a table describing what each test in tests.sh catches, the prerequisites, and how to override $BSCP. Kept a condensed manual snippet for interactive debugging. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bscp: three behavior changes plus one bug fix:
- new parse_block_count() helper. Bare integer = block count (back-compat); K/M/G/T suffix = byte size, encoded as a negative sentinel until blocksize is known.
- -B/--block-count now uses parse_block_count as its argparse type, and the help text documents both forms plus the implicit --allow-truncate.
- main() resolves the byte-suffixed form to a real block count once blocksize is settled, then sets args.allow_truncate = True whenever block_count > 0.
- do_sync() keeps the original local size as local_size_actual, and after both sides' sizes are known emits a Warning:
-B requests up to X but {local|remote} source has only Y message when start_offset + block_count * blocksize exceeds the actual source size.
- fixed typo: the resume-command builder emitted -n N instead of -B N, which would have been an invalid flag if a -B run hit a connection failure.
tests.sh: three new tests (suffix support, implicit --allow-truncate, overshoot warning).
CLAUDE.md: updated the size-validation invariant note and the test table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tests - truncate check now compares dst_size against eff_src = min(src_actual_size, requested_B_limit). Overshoot (-B exceeds source) is a warning; dst < eff_src is still an error unless --allow-truncate is explicit. - wire-side ALLOW_TRUNCATE is set whenever -B is in effect, so the remote's own truncate check (which only sees the wire-capped size) doesn't second-guess. The client's check, which sees actual sizes, is authoritative. - remote's section loop wrapped in try / except (struct.error, IOError, OSError) so a client-side abort post-handshake terminates the remote cleanly instead of leaking a traceback to the user's stderr. - Removed the args.allow_truncate = True implication in main. tests.sh: - Renamed test_block_count_implies_allow_truncate → test_block_count_pull_no_truncate_needed to match the actual behavior (no flag implication; just no false trigger). - Added test_block_count_truncate_still_required for the new explicit-error case (push -B 8M to a 4M dst refuses without --allow-truncate, succeeds with it). CLAUDE.md — invariant section and test table updated to spell out that -B and --allow-truncate are independent. All 16 tests pass against ./bscp. Rebuilt the Nuitka binaries to pick up the protocol-side changes (the remote's exception-wrapper is in remote_script). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
.github/README.md: - Destination preamble — clarifies that the size requirement is against the requested size (source, or -B limit), and points at --allow-truncate as the override. - Options table — added --allow-truncate and --buffer (both were missing entirely), and rewrote the -B row to mention K/M/G/T suffix support and the overshoot warning. Removed the misleading "destination will not be fully updated; use intentionally" warning, since -B no longer silently truncates. - Examples — added a -B 1G example and a --allow-truncate example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- bscp (fail() closure in do_sync): also close proc.stdout before proc.wait(). Previously, when -B was used with a destination smaller than the source (and no --allow-truncate), the client set the wire-side ALLOW_TRUNCATE bit so the remote bypassed its own check and started streaming Phase-A hashes; once that stream filled the OS pipe buffer, the remote blocked on stdout.write() while the client blocked on proc.wait() — classic deadlock. Closing stdout unblocks the remote with EPIPE, its existing try/except catches it and exits cleanly. - tests.sh: regression test using sparse 256 MiB src / 128 MiB dst with -B 512M and a 30s timeout safety net. - CLAUDE.md: updated the fail() description to document why both pipes are closed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…atch exit resume status limitation - New parse_resume_from() accepts byte sizes (as before) plus NN% / NN.N% (0–100), encoded as a negative permille sentinel. - -r/--resume-from argparse uses the new parser; help text updated. - main() resolves the percentage against os.path.getsize(local_file) before section-boundary rounding. If the file is missing, falls through to the existing friendly error in do_sync. - --batch help text now documents the exit-code-only contract and points callers needing resume at --quiet. tests.sh - New test_resume_from_percent (10 MiB file, -r 50% with -s 2M, mid-file diffs). - All 18 tests pass (17 prior + 1 new). CLAUDE.md - New row in the Tests table. - Note in Quiet/batch section explaining the batch+resume limitation and the percentage's local-file-size base (and the truncate edge case). No protocol changes; the wire-side start_offset is still always a byte value rounded to a section boundary. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…Nuitka binaries will still work
bscp
- New remote_perl triple-quoted string (~110 lines, compact style matching remote_script).
- build_ssh_cmd() now emits a for py in python3 python2 python; ...; command -v perl && exec perl -e 'eval pack(qq{H*}, q{HEX})' wrapper.
- BSCP_FORCE_PERL=1 env var skips the Python branch — purely a test hook, transparent in normal use.
- Fallback failure prints bscp: no python or perl found on remote and exits 127.
tests.sh
- test_perl_remote_push and test_perl_remote_pull exercise the Perl path under BSCP_FORCE_PERL=1.
Both auto-skip with command -v perl >/dev/null || return 0 so the suite stays green on hosts without Perl.
CLAUDE.md
- Architecture diagram lists remote_perl.
- New "Perl fallback" section explains the hex-pack-eval encoding, version requirements (Perl 5.10+),
the bilingual maintenance constraint (protocol changes touch three places now), and the \\n/\\&/\\z doubling rule for Python's triple-quoted container.
- Nuitka section's remote-requirements line now includes perl.
- Two new test rows.
- New "Documentation map" section right after Project overview, listing where to update what when functional behaviour changes
(.github/README.md, CLAUDE.md, PROTOCOL.md, index.html) and noting that bug fixes without behavioural change don't need doc updates.
.github/README.md
- Intro line now says "embedded in the client and executed via python(/2/3) -c over the SSH connection, with a Perl fallback for hosts that have no Python interpreter."
- Requirements: "Python 3 on the local host and Python 2/3 or Perl 5.10+ on the remote host."
- -r row updated to OFFSET metavar and mentions NN% / NN.N% form.
- --batch row gets a second line: "Cannot convey a resume offset — use -q instead if a caller needs to parse the 'Resume with:' stderr line."
- New example: bscp -r 50% /dev/sda myhost:/dev/sda.
The wire protocol is unchanged; the Perl remote is byte-for-byte compatible with the existing Python remote.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- do_sync() parameters: local_filename → local_file, remote_filename → remote_file. Now consistent with main(). - open_mode line: 3-line comment explaining that local opens 'rb+' on pull (destination) and 'rb' on push (source), and that the remote does the symmetric inverse. - Size variables: local_size_actual → local_size (the raw stat result now has the natural name); the -B-capped value sent in the header is now wire_size. The two surrounding comment blocks were updated to use the new names; the truncate check (local_size) and wire-side computations (wire_size) now read more clearly because each variable's role matches its name. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…lt --retries 3 - build_ssh_cmd(): append `-o ServerAliveInterval=15 -o ServerAliveCountMax=4` after the user's -o options. A silently-dropped TCP connection (router NAT timeout, peer kernel stall, ISP blip) now surfaces as BrokenPipeError within ~60s instead of hanging forever, which engages the existing ConnectionLost / --retries path. User -o wins because ssh applies the first matching -o. - remote_script + remote_perl pull phase B: flush stdout per block instead of once per section. No-op at the default 64 KiB blocksize (each write already bypasses Python's 8 KiB BufferedWriter and goes straight to the pipe), but eliminates a latency-hiccup class where, with small custom -b sizes, blocks would sit in the remote's I/O buffer waiting for buffer-fill — a stall indistinguishable from a deadlock under load. - --retries default changed from 0 to 3 so transient network failures (now detected within ~60s thanks to the keepalive change above) auto-recover out of the box. New DEFAULT_RETRIES module-level constant; build_resume_cmd() compares against it so resume hints no longer carry a redundant `--retries 3`. Pass `--retries 0` to opt out. - .github/README.md and CLAUDE.md updated to document the keepalive default and its override precedence, the new --retries default, and the per-block-flush rationale (incl. the cost analysis showing it's free at the default blocksize). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…og + write-side mirror - import select; new DEFAULT_IO_TIMEOUT = 0. - New IOTimeout(OSError) exception class. - IOCounter constructor takes timeout=; when > 0 grabs raw fds and switches read/write to select.select + os.read/os.write loops. Default (0) path unchanged → byte-for-byte same buffered I/O. - do_sync(... io_timeout=0 ...); Popen(... bufsize=0 ...) only when watchdog active. - IOTimeout added to the per-section except tuple; closes both stdin AND stdout before proc.wait() (same rationale as fail()). - New CLI flag --io-timeout SECS (float, default 0). - build_resume_cmd forwards --io-timeout so a resumed run keeps the setting. Docs - .github/README.md — new options-table row. - CLAUDE.md — architecture-diagram entries for IOCounter (raw path) and IOTimeout. Verification - Full tests.sh: 20/20 pass (default path). - Smoke push 20 MiB --io-timeout 30 -s 4M: md5 match. - Smoke pull 20 MiB --io-timeout 30 -s 4M: md5 match. Note No tests.sh entry for actually triggering the timeout — mocking a hung remote needs a stub ssh on PATH. Out of scope for now, awaiting real-world test results. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New format:
"copy WPOS/TOTAL (PCT%) block C/N (SEC_PCT%) (SPEED KiB/s) ELAPSED (ETA)"
- WPOS = current block write offset (start, never hits 100% by design)
- C/N = cumulative blocks copied / cumulative diffs found so far (moving target)
- SEC_PCT = blocks-done-this-section / section diff count
- Dropped [CUR/TOTAL] section prefix and cur_section/total_sections vars (unused after change).
- Added total_blocks_copied cumulative counter; section_diff_count captured per section.
- enumerate() on push iterations; explicit blocks_done counter on pull.
Docs
- CLAUDE.md "Progress format" block updated for both scan + new copy layout.
- Also fixed pre-existing scan-doc drift (diff=N → diff blocks=N, "blocks" → "bytes").
Verification
- Regression suite: 20/20 pass.
- Live push frame: copy 149M/2048M (7.3%) block 965/1234 (35.6%) (285388 KiB/s) 0:00 (0:04).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: smoothing prev_eta got initialized from the bootstrap estimate during ~ phase. When confident flipped True, floor clamp inherited the inflated bootstrap value and refused to drop faster than 1s/wall-second. Fix: added eta_have_real_sample flag; first confident tick re-anchors prev_eta to the real raw value. Subsequent confident ticks engage floor clamp normally. Before/after (4GB fresh run, real wall-time ~6s): - before: bootstrap ~8:12, then 8:11, 8:11, 8:10, 8:09 … (decaying 1s/tick) - after: bootstrap ~5:11, then 0:07, 0:08, 0:07, 0:07, 0:06 … (tracks reality) Resume -r 50% (real ~3s): bootstrap ~3:25:28 → 0:03 on first confident tick → converges. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Changes: - import math added. - scan_rate / copy_rate are now time-weighted EMAs with τ=5s. Seeded from cumulative average at first sample where phase_secs ≥ RATE_RELIABLE_SECS (or written ≥ 4 MiB for copy), then updated each progress tick with w = 1 − exp(−dt/τ). - Copy EMA also updated during scan ticks (uses cumulative total_copy_time / total_written) so confidence promotes even between sparse copy phases. - Floor clamp removed entirely. smoothed_eta() deleted. Raw ETA displayed. - confident redefined as "rate EMAs are seeded" — controls ~ marker only. Behavior: - 4GB fresh run (real ~7s): ~5:14 → ~0:07 → ~0:07 → 0:05 → 0:06 → 0:04 → 0:05 → 0:03 … — confidence promotes at tick 4, ETA tracks reality. Drops faster than 1s/tick when measurements improve. - 4GB dry-run (real ~7s): ~7:30 → ~0:06 → 0:05 → 0:05 → 0:04 → 0:04 … — converges in 3 ticks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-snap every 5s (ETA_REFRESH_INTERVAL), counts down 1s/s between snaps. Bootstrap window unchanged (still tracks every tick). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bscp.python2 - Py2 shims:
- shebang #!/usr/bin/env python + # -*- coding: utf-8 -*-
- from __future__ import division
- binascii.hexlify(...).decode('ascii') (no bytes.hex())
- try/except shlex.quote → pipes.quote (named _shquote)
- _PIPE_ERRORS tuple via try/except for BrokenPipeError/ConnectionResetError
- super(ConnectionLost, self) form
- (IOError, OSError) on local file open
- single-element list-cell shims (var[0]) replacing nonlocal in display_eta, update_scan_rate_ema, update_copy_rate_ema, show_copy_progress — 9 cells total
- raise X instead of raise X from Y
Dropped from Py2 build: --io-timeout flag + its select/raw-fd path + IOTimeout class. Memoryview slicing into os.write in Py2 is fragile;
SSH ServerAliveInterval=15 keepalive (kept) gives ~60s dropped-connection detection, which is good enough for the fallback.
New ETA model, EMA rates, damping, keepalive, summary "diff blocks=" — all ported.
CLAUDE.md changes:
- Doc-map paragraph: note bscp.python2 is milestone-only, not per-commit.
- New section "Python 2 client fallback" between "Nuitka builds" and "Testing": purpose, maintenance policy, feature parity (drops --io-timeout only), shim table, update procedure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…alculated value at ETA_REFRESH_INTERVAL
- Shebang reverted to #!/usr/bin/env python + utf-8 coding cookie.
- from __future__ import division added.
- binascii re-imported; select/shlex dropped (no --io-timeout).
- _shquote shim (Py3 shlex.quote / Py2 pipes.quote).
- _PIPE_ERRORS tuple shim (no BrokenPipeError/ConnectionResetError on Py2).
- IOTimeout class removed; IOCounter collapsed to buffered-only path (no raw-fd _read_raw/_write_raw, no timeout arg, no popen_bufsize).
- DEFAULT_IO_TIMEOUT constant + --io-timeout argparse entry + io_timeout= kwarg removed from do_sync and call site.
- super(ConnectionLost, self).__init__(...) form.
- except (IOError, OSError) as e on local-file open.
- binascii.hexlify(remote_perl.encode('utf-8')).decode('ascii') for the Perl hex blob.
- _PIPE_ERRORS in the connection-loss handler; raise ConnectionLost(last_sec_start) (no from exc).
- List-cell shims for all closure-mutated variables: t_last_progress, eta_smoothed, eta_displayed, eta_displayed_at,
ema_scan_rate, ema_copy_rate, rate_prev_scan_secs, rate_prev_scanned, rate_prev_copy_secs, rate_prev_written.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…0) at the end for consistency
…ata, properly signalling and handling a lost connection earlier
… reads The remote script relied on struct.unpack raising struct.error to detect short reads of the 8-byte length fields, but the push phase-B data-block read (f.write(stdin.read(bl))) had no such guard: a dropped SSH connection mid-block returned fewer bytes than requested, and f.write silently wrote a truncated block to the destination, corrupting it. Add an rd(n) read-exactly helper (loops until n bytes or raises EOFError) mirroring the Perl remote's r() helper and the client's own short-read guards, and route every remote stdin read through it. On a blocking pipe a short read means EOF, i.e. the connection is gone, so signalling it is always correct. EOFError is not an OSError subclass on Python 2, so it is named explicitly in the handshake try/except and the section-loop except. The Perl remote already guarded every stdin read via r(); no change needed there. Also refreshes bscp.python2 to parity (milestone update): the shared remote_script string is byte-identical to bscp's, so the same fix lands in both. No client-side Py2 shim sites were touched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scan phase is single-core-bound on fast (NVMe/local) storage: one core computes digests serially while the disk sits idle. Fan the per-block hashing across a thread pool on both client and remote. hashlib releases the GIL for blocks >= 2048 bytes, so threads give true parallelism at the default 64 KiB block size — no multiprocessing, no IPC. The wire contract is digest order only, so each side picks its thread count independently: no protocol/header change, no negotiation. python3-only on both ends; python2, Perl, and the bscp.python2 client stay single-threaded. - New remote_script_mt: python3 threaded twin of remote_script, hex-encoded and run via exec(bytes.fromhex(...)), so the no-"/no-$/no-% rules don't apply. build_ssh_cmd dispatches python3 -> MT, python2/python -> legacy remote_script, perl -> remote_perl. - Client phase A: order-preserving bounded feed/drain pipeline (hash_window = 2x workers), memory capped independent of section size. - --hash-threads N (0 = auto: min(cores, 4)), forwarded into resume command. - New BSCP_FORCE_PYTHON2 test hook: exercises the single-threaded legacy remote_script even on a python3 host (previously never reached by tests). - tests.sh: +4 tests (threaded N=4/N=1, legacy remote push/pull). 24 pass. - Docs: CLAUDE.md (architecture, 4 protocol-sync sites, new section), README options table, bscp header. Measured: 600 MiB localhost scan, both ends hashing, 3.6s -> 1.9s (N=1->4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Indent the three embedded remote sources (remote_script, remote_script_mt, remote_perl) with tabs instead of 4 spaces. Two are hex-encoded onto the ssh command line; one tab replaces a 4-space unit, roughly halving leading- whitespace bytes on the wire (deep nesting saves most). Client code stays 4-space — tabs live inside string literals only, so no TabError and the file's own indentation is unchanged. Added a ts=4 vim modeline and a note. Append a literal `bscp-remote` argument to each exec'd remote (python3/MT, python2/legacy, perl). The bodies never read argv, so it is inert, but it shows in the remote command line: `ps aux | grep bscp-remote` locates the helper on the destination host. Docs: CLAUDE.md (tab constraint, process-marker section, encoded sizes), README (multi-thread note + ps tip). Verified: legacy compiles py2+py3, MT py3, perl -c OK, tabs survive the hex round-trip, marker live-visible in ps. 24/24 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md tripped the >40K context warning. Leading whitespace was only 299 bytes, so tabifying would not have helped; the bulk is prose. Move five deep-dive sections into docs/ and keep CLAUDE.md as the orientation + invariants + testing index: docs/remote-execution.md remote script constraints, bscp-remote marker, multi-threaded hashing, Perl fallback docs/protocol-internals.md symmetric size validation, PULL_WINDOW docs/eta-model.md unified scan+copy ETA model docs/python2-client.md bscp.python2 shims and policy docs/nuitka.md prebuilt binaries and rebuild Add a "Subsystem deep-dives" index and a Documentation-map entry; fix all cross-references (in-file anchors -> docs/ paths, bscp Perl-fallback comment). No content lost; links verified; bscp still compiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document real-world use cases (off-site disk imaging with overlayroot GHOST mode, SSD-friendly incremental backups, compressed/sparse/mountable images) in README and index.html. Surface portability as a USP — no remote install (python3/python2/perl), single-file Nuitka binary, bscp.python2 fallback — linking the docs/ deep-dives. Fix index.html: replace the stale positional-syntax usage with the current `bscp [options] SRC DST` overview (full option list deferred to README), and close an unconforming <li> so it validates. Rewrite docs/nuitka.md as a generic build guide (binaries are gitignored, not shipped): venv setup, the onefile command, no-cross-compile caveat, and both motivations — no client Python, and embedding a newer/faster CPython. Drop "prebuilt/checked-in bscp.amd64/arm64" wording everywhere. Mark the repo as a fork: credit the original bscp (bscp-tool/bscp, vog/bscp) in the bscp header, README, index.html and CLAUDE.md; retain the ISC notice; point self-references at a placeholder fork URL (TBD). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…conds, carrying to minutes, but not hours
Reorganize argparse into ordered groups (transfer tuning, performance, SSH connection, resilience and resume, safety, output) so related flags sit together in --help. -a/--algorithm now lists the portable set (md5, sha1, sha224, sha256, sha384, sha512 — work on python3/python2/Perl remotes) plus the local hashlib extras, with a note that extras need a Python remote. Added a parse_algorithm() arg guard rejecting unknown algorithms and zero-digest XOF functions (shake_*) that would break the fixed-digest wire protocol. Docs: README -C row and index.html note SSH compression as a WAN USP (speeds bandwidth-limited links, slows fast LANs). Fork URL set to https://github.com/groenewe/bscp (TBD removed). README/CLAUDE/ remote-execution docs updated; new regression test for the algo guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sts under py2 Bring bscp.python2 up to parity with bscp, carrying every milestone change since the last refresh except the two python3-only client features. Ported: - remote_script_mt (verbatim, byte-identical) and the python3->MT dispatch in build_ssh_cmd, so a python3 remote still hashes multi-threaded when driven from the Py2 client (thread count baked as _remote(0) = remote auto-detect; the Py2 client has no --hash-threads value to pass). - bscp-remote process marker and BSCP_FORCE_PYTHON2 test hook. - Tab-indented remote source literals + ts=4 modeline. - Hash-algorithm advertise/validate (PORTABLE_ALGOS, algorithm_help, parse_algorithm; -a now validated) and grouped --help layout. - Fork attribution header. Deliberately not ported (multi-threading is not a Py2 requirement): - --hash-threads option and the client-side ThreadPoolExecutor pool; client hashing stays serial. - --io-timeout watchdog (IOTimeout, raw-fd select paths, bufsize=0). tests.sh: add --force-all and auto-skip three tests when $BSCP runs under a Python 2 interpreter (detected via shebang + python -V): the two --hash-threads tests (option absent) and the -a rejection test (Py2 hashlib lacks the shake_* XOF functions it probes). Summary line now reports skipped count. Verified: py2/py3 py_compile clean; remote strings sha256-identical to bscp; BSCP=./bscp.python2 ./tests.sh = 22 passed, 3 skipped; ./tests.sh = 25 passed. Docs: docs/python2-client.md and CLAUDE.md updated for the kept-but-rewired MT remote, the ported features, and the new skip/--force-all behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-T, python3 only)
The header comment referenced the private "main-HG" branch, which is not published on the fork. Reword to a fork-relative attribution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…--quiet (--batch remains totally silent)
…rato.de VM root device name is /dev/vda
Handshake-stage failures raised a plain RuntimeError, caught by
do_sync's `except RuntimeError` (exit 1) before `except ConnectionLost`
(retry) could engage. When ssh failed to reconnect ("No route to host"),
the remote sent no sanity digest, io.read got EOF, and bscp died with
"Remote script failed to execute" — never retrying, even though the link
had been up on prior attempts.
Add a lost() helper that raises ConnectionLost(start_offset) so --retries
re-establishes the connection. Distinguish a genuinely fatal config error
(SSH wrapper exit 127, "no python or perl found on remote") from transient
loss via proc.returncode: 127 stays a hard RuntimeError, anything else
retries. Wrap the handshake writes too, so a pipe break mid-header routes
through the same path. "Remote file not found" stays fatal.
bscp.python2: port the same fix, but the applied handshake clause used
literal BrokenPipeError/ConnectionResetError (py3-only names → NameError
on py2, masking the real pipe error) plus IOTimeout (a watchdog feature
the py2 client lacks). Replace _PIPE_ERRORS with named _BrokenPipeError /
_ConnectionResetError aliases (real classes on py3, IOError/OSError on
py2) so each except clause can list exactly the exceptions it needs;
drop IOTimeout from the handshake clause.
Add regression test "connection failure engages retries, exits 3"
(unreachable RFC5737 host) and document it in CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Limit network I/O to RATE bytes/sec (K/M/G/T suffix), governing both directions combined. Client-side only: a token bucket in IOCounter sleeps to hold throughput at the rate, with the burst budget capped at one second's worth. Push throttles write() directly; pull is throttled via read() backpressure on the SSH pipe. No protocol or remote-script change. Bytes are counted pre-compression (upstream of ssh -C), so the limit is an upper bound on wire usage: exact for incompressible data (the raw block-device case), conservative — wire below RATE — for compressible. The token bucket carries a negative balance forward rather than zeroing it after sleeping; the next refill, sized by the elapsed sleep, repays the deficit exactly. Zeroing would double-credit and let the rate run ~2x over the cap. Ported to bscp.python2 as well (plain arithmetic, no raw-fd path), so the Py2 fallback keeps feature parity; of the resilience-group flags only --io-timeout remains dropped there. Docs: README options row + example + -C nuance; CLAUDE.md IOCounter blurb and testing table; python2-client.md parity note. New regression test asserts the throttle engages via a timing floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just a pointer to my forked repo. Feel free to merge, create a new branch, add a pointer to my repo, or do nothing.
Thanks for the inspiration! I've tested my new version quite extensively, added functionality and fixed a few bugs.
AI assistance does mean getting a lot of code and doc bloat, though, but gave me a lot of flexibility and speed.
Beta quality imho. YMMV :-)