perf(backup): cut Azure container/list ops in rclone sync#106
Merged
Conversation
Add --fast-list and --azureblob-no-check-container to both rclone backup command templates. Azure's "List and Create Container Operations" meter was the single largest line on shardbackupstorage (~3x the actual data-stored cost, ~€15/mo). Two causes: - Without --fast-list, rclone lists the destination hierarchically, issuing one List Blobs op per subdirectory. user_data trees are deep, so this scales with folder count x shards x nightly runs. --fast-list does a single recursive listing (~1 op per 5000 blobs) instead. - Without --azureblob-no-check-container, rclone issues a create/check container op on every invocation (2x per shard per night). The controller already creates the container when minting the SAS, so the check is pure waste. Read-side/listing change only; sync semantics unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the
shardbackupstorageAzure account, the "List and Create Container Operations" meter is the single largest cost line — roughly 3x the actual data-stored cost (~€15/mo), ~70% of the account's spend. Nightly-per-shard cadence can't reach that op volume from container creates alone; it's List Blobs ops fromrclone sync.Root cause
shard_core/service/backup.pyrunsrclone synconce per backup directory (core,user_data) per shard, nightly (0 3 * * *):--fast-list→ rclone lists the destination hierarchically, one List Blobs op per subdirectory.user_data(app data) is a deep tree, so op count scales with folder-count × shards × 30 nights. This is the bulk of the ~3M ops/month.--azureblob-no-check-container→ rclone issues a create/check-container op on every invocation (2× per shard per night). The controller (get_backup_sas_url) already creates the container when minting the SAS — the check is pure waste.Fix
Add both flags to
COMMAND_TEMPLATEandCLEARTEXT_COMMAND_TEMPLATE:--fast-list— single recursive listing (~1 op per 5000 blobs) instead of one-per-directory. Modest extra RAM during listing; blob counts are small.--azureblob-no-check-container— skip the redundant container check/create.Read-side / listing change only —
syncsemantics unchanged.Expected impact
Drops the dominant cost line by most of its value → total subscription run rate ~€41/mo → ~€26/mo.
Test notes
No tests assert the rclone command string.
just cleanup(ruff/black) could not run — both absent from the env; the change is whitespace inside an existing triple-quoted string, so formatting is unaffected. Verified the new flags tokenize as distinct argv entries undercommand.split().🤖 Generated with Claude Code