Support repeated --glob/--exclude flags on download/delete/list#769
Merged
Conversation
Previously --glob accepted a single string and split on `|` to form
multiple patterns. Users would naturally reach for `--glob A --glob B`,
which argparse silently truncated to the last value -- files were
missed without warning. This change accepts all four forms consistently:
--glob "a" -> match a
--glob "a|b" -> match a or b (unchanged)
--glob a --glob b -> match a or b (new)
--glob "a|b" --glob c -> match a, b, c (new)
The same applies to --exclude on `ia download` and --glob on
`ia delete` / `ia list`.
Implementation:
- Added _flatten_pipe_patterns() in item.py to normalize a glob arg
(str or list[str], elements optionally `|`-separated) into a flat
list. Item.get_files() now uses it for both glob_pattern and
exclude_pattern, so API callers passing mixed forms like
glob_pattern=['*.mp4|*.xml', '*.jpg'] now work.
- Switched the three CLI flags to `nargs=1, action="extend"` -- the
same pattern already used by --format, --source, --exclude-source.
args.glob / args.exclude are now list[str] | None; downstream code
was already accepting both shapes via Item.get_files().
- ia_list's local pipe-split was updated to flatten the new list shape
via itertools.chain.
Tests cover the four call styles in tests/cli/test_ia_download.py
using the in-process ia_call + IaRequestsMock + --dry-run pattern
(offline). API tests in tests/test_item.py extend
test_get_files_by_glob{,_with_exclude} to cover the new mixed-form
inputs.
Also includes carried-over 5.9.0 release scaffolding (changelog
entries for #753, #767, #768 and version bump to 5.9.0.dev1) that was
present in the working tree before branching from master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the "patterns may also be combined with `|`" clause from the --glob and --exclude help strings. Repetition (`--glob a --glob b`) is the discoverable form; the pipe form keeps working but is documented in docs/source/cli.rst rather than the per-flag help. Also normalize "files whose filename matches" -> "files matching" on `ia download` to match `ia delete` and `ia list`. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanups from the review: 1. Promote `flatten_pipe_patterns` to `internetarchive/utils.py` (was a module-private `_flatten_pipe_patterns` in item.py) so the CLI can reuse it. `ia_list.filter_files` now calls the shared helper instead of duplicating the `chain.from_iterable(...)` algorithm inline. 2. Widen `glob_pattern` / `exclude_pattern` annotations from `str | None` to `str | list[str] | None` everywhere they were typed -- `Item.download()` and the `api.py` wrappers (`get_files`, `download`, `delete`). The runtime already accepted both shapes via `Item.get_files()`; only the annotations and docstrings were stale. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ia download,ia delete, andia listnow accept--glob(and--excludeondownload) repeated, pipe-separated, or mixed — all four forms produce the union of patterns. Existing single-flag and pipe usage is unchanged.Item.get_files()now pipe-splits list-element patterns too, so API callers can pass mixed forms likeglob_pattern=['*.mp4|*.xml', '*.jpg'].--glob "a"--glob "a|b"|)|)--glob a --glob bb--glob "a|b" --glob ccImplementation notes
_flatten_pipe_patterns()helper initem.pynormalizes a glob arg (str, list, or mixed) to a flat list.Item.get_files()uses it for bothglob_patternandexclude_pattern.nargs=1, action="extend"— same pattern as--format.args.glob/args.excludeare nowlist[str] | None; downstream code already accepted both shapes viaItem.get_files().ia_list's local pipe-split was updated to flatten the new list shape viaitertools.chain.This PR also carries forward release scaffolding that was sitting uncommitted in the working tree before branching: changelog entries for #753/#767/#768 and the
5.9.0.dev1version bump.Test plan
pytest tests/test_item.py tests/cli/test_ia_download.py tests/cli/test_ia_list.py tests/cli/test_ia_delete.py— 63 passedpytest tests/(excluding live-network tests) — 239 passed, 37 skippedruff checkclean on all touched filesia download --helprenders the new help texttest_glob,test_exclude) continue to pass on CI — they exercise back-compat for the single-flag and pipe forms.🤖 Generated with Claude Code