Conversation
Owner
ajslater
commented
Apr 29, 2026
ajslater
commented
- Update to Confuse 2.2.0
- Introduce new PicoptSettings class to type the config.
commit 8f869a7edfcce5026f442ba0cb4b3ad5483848c9
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 16:24:33 2025 -0700
ignore obeys filesystem case sensitiviy
commit 749a4a9e59505c2e71970bc00e531548d614a946
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 16:24:06 2025 -0700
fix adding timestamps from inside archives
commit f40ac06f23061d77945f1aede85719e496b5c4a4
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 15:55:28 2025 -0700
move WalkSkipper back into walk dir
commit bfcf32ae57c669f236e31c2b08c542b2059f1447
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 15:54:11 2025 -0700
move walk out of __init__.py
commit b2dcea5f22e4d6fd2442b8fabd04351ea79563c1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 15:51:03 2025 -0700
simplify using timestamps inwalk and archive. remove cruft
commit e4afb1692f34d6d30b74cd5f2aea56bf4e2a340e
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 15:37:26 2025 -0700
organize news
commit aea9ba29dbd14f0ea37211ed75585ed1b0fbfd4b
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 24 15:35:20 2025 -0700
treestamps can skip archive without opening them
…ave memory repacking animated
archive gets an _optimize_in_place_on_disk flag container copy_skipped_files turns into a hydrate_optimized paths zip deletes files it would replace on disk and closes the file before appending the new files.
* bump version and news * rightyper typechecked * Fix rightyper-introduced type annotation errors Fixes 68 basedpyright errors and 51 ty errors introduced by the automated rightyper pass. Changes are annotation-only. - PathInfo.__init__ path_info parameter widened from None to PathInfo | None - Removed spurious self: "picopt.xxx.Y" annotations naming wrong modules - Simplified **kwargs bogus unions to Any for variadic super().__init__ passthroughs - config/handlers.py handler_stages/convert_chain/native types widened to proper base types - scheduler.py _trigger_fail_fast/_cancel_subtree widened Exception to BaseException | None - detect_format.py filtered image.info keys to str; suppressed dynamic mpinfo attr - pil_convertible _file_format image_class widened from BmpImageFile to Image - tar.py removed type[TarGzDetector] cls annotation violating LSP - tests: PosixPath -> Path; ts:None/config:None -> float|None/dict|None - Cleaned unused TYPE_CHECKING imports left behind after removing bad annotations Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * correct typing changes * revert to python 3.10 and remove righttyper * update devenv * remove quoted self types * remove self types * format & fix --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix pickling of ContainerHandler when a timestamps file exists When a prior picopt run leaves a .picopt_treestamps.yaml, loading it via ruamel.yaml leaves the YAML instance holding a Reader with an open BufferedReader. Submitting an UnpackJob (or RepackJob) to the ProcessPoolExecutor then fails with "cannot pickle 'BufferedReader' instances" because the handler carries that Grovestamps through its _timestamps / _skipper attributes. Add __getstate__ on ContainerHandler that nulls both fields before pickling. The scheduler owns the real Grovestamps on the main thread; workers never actually used it (in-archive timestamp consumption happened in a pickled copy whose mutations never propagated back). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * remove clean_for_repack, superseded by __getstate__ The method nulled the same two fields __getstate__ now nulls at pickle time, and its one caller always ran on a handler that had already round-tripped through a worker — so the fields were already None. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
commit 83a5253785fccc471a6dbd75b4d1eba3074c9e8c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 29 02:18:19 2026 -0700
bump version and news
commit 9b03aed
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:44:04 2026 -0700
update treestamp loggins
commit 1b7bdc7
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:32:59 2026 -0700
fix implementation of timestamp loggins to log after not before
commit 8d25937
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:25:28 2026 -0700
Log timestamp load/dump and surface INFO at default verbose=1 (#111)
Treestamps 4 dropped its own load/dump prints, and picopt's new logger
mapped INFO to verbose>=2, so default runs went silent for messages the
old termcolor Printer always showed (config-style force_verbose=True).
- _VERBOSE_LEVEL: bump so verbose=1 (the argparse default) emits INFO,
matching the old printer's "force_verbose" tier.
- walk.py: log "Loading timestamps for: …" before Grovestamps init and
"Dumping timestamps for: …" before dumpf(). INFO renders cyan via the
existing LEVEL_STYLES.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 4415376
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 18:41:23 2026 -0700
Pre-walk file count for a determinate progress bar (#110)
* Pre-walk file count for a determinate progress bar
Without a total the bar showed only an indeterminate spinner + count.
Add ``Walk._count_total`` that mirrors ``walk_file``'s recursion gate
(symlinks, timestamp filenames, ignore patterns, recurse flag) so each
non-recursing visit contributes one mark — matching the events the
scheduler dispatches through Reporter — and pass it as ``total=`` to
``make_progress``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use os.scandir for the pre-walk count
DirEntry caches ``is_dir`` / ``is_symlink`` from the directory listing,
so deep trees skip an extra ``stat`` per entry — meaningful on slow or
network filesystems.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 6a46d43
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:48:59 2026 -0700
Replace termcolor Printer with loguru + rich logger (#109)
The old Printer class wrote dots and messages directly to stdout from
both the main process and worker processes. This change discards it for
a centralized logger modeled on nudebomb's progress branch:
- New picopt/log/ package: shared rich Console, loguru sink, a streaming
CharStreamColumn progress bar, a Stats + render() summary, and a
Reporter that bundles them and dispatches each ReportStats outcome.
- All call sites converted: Printer.{saved,converted,lost,error,skip,
warn,config,...} → logger.* and progress.mark_*. Worker-side dot and
lifecycle calls dropped — workers can't reach the parent's live region,
so per-file progress is now driven from the scheduler when each result
comes back.
- Centralized MARKS table in picopt/log/styles.py drives the streaming
chars, the loguru sink colors, the summary table row colors, and the
--help epilogue legend, so the same outcome reads identically
everywhere. Style choices mirror the old termcolor palette so longtime
users see the same colors for the same outcomes.
- Scheduler now takes a Reporter; Totals removed. report.py is a pure
data class (ReportStats) with no printer dependency.
- doctor.py and the cli help epilogue rewritten with rich (rich.markup
escape() for path strings).
- Grovestamps now constructed with verbose=0 so treestamps's internal
printer doesn't bypass the rich Live region.
- loguru~=0.7 added; termcolor dropped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 03cdcb0
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:34:02 2026 -0700
move set_jpeg_xmp into jpeg plugin
commit b283374
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:06:50 2026 -0700
update devenv. treestamps 4, rich 15
) (#115) Confuse 2.2.0 made ``AttrDict`` a properly-parameterized generic; an unparameterized ``AttrDict`` resolves to ``AttrDict[str, object]``, so every downstream access — ``cfg.bigger``, ``int(cfg.jobs)``, ``cfg.paths`` flowing into ``Iterable[str]``, etc. — now reads as ``object`` and stops type-checking. We had ~31 basedpyright errors after the bump, all rooted at AttrDict consumers. Fix: keep using confuse for what it's good at — YAML / env / CLI parsing, type coercion, requiredness — and convert the validated ``AttrDict`` into a typed frozen dataclass once. Every downstream module then takes ``PicoptSettings`` instead of ``AttrDict``. - ``picopt/config/settings.py``: new ``PicoptSettings`` (and nested ``ComputedSettings`` / ``IgnorePatterns``) frozen dataclasses mirroring the existing ``MappingTemplate`` schema. ``Sequence(...)`` fields are ``tuple[X, ...]`` because the dataclass is frozen. ``handler_stages`` is typed ``dict[Any, Any]`` to avoid an import cycle with ``picopt.plugins.base``; consumers know its concrete shape. - ``picopt/config/__init__.py``: ``PicoptConfig.get_config`` now returns ``PicoptSettings``. New ``_settings_from_attrdict`` does the field-by-field copy (typed ``Any`` so it can read AttrDict attributes directly — the conversion *is* the schema escape hatch). - All consumers (``picopt/path.py``, ``picopt/report.py``, ``picopt/walk/*``, ``picopt/plugins/base/{handler,container}.py``) replace ``from confuse(.templates) import AttrDict`` with ``from picopt.config.settings import PicoptSettings``. - ``Walk._init_timestamps`` builds a shallow ``Mapping[str, Any]`` for Grovestamps' ``program_config=`` so we don't have to serialize the whole frozen dataclass (especially the computed sub-fields, which carry ``re.Pattern`` and class-keyed ``dict`` objects). - ``is_path_ignored`` switched from ``ignore and bool(...)`` to ``ignore is not None and bool(...)`` — its return type is now honestly ``bool``, not ``bool | re.Pattern | None``. - ``Walk._dump_timestamps`` joins the dumpf return paths via ``str(p)`` (treestamps' ``dumpf`` returns ``tuple[Path, ...]``). pyproject pin: ``confuse~=2.1.0,<2.2.0`` → ``confuse~=2.2.0``. Verification: ruff clean, basedpyright clean (0 errors, was 31), ``uv run pytest --deselect …test_timestamp_parents`` 149 passed (deselected test is a pre-existing treestamps v4 regression), ``radon cc`` all A/B, CLI smoke run with env var + CLI flag still layers correctly. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two warnings flagged by `make typecheck complexity`:
- ``picopt/config/__init__.py``: confuse 2.2.0 already types
``config.get(MappingTemplate)`` as ``AttrDict[str, object]``, so the
defensive ``isinstance(ad, AttrDict)`` check after it was statically
always-true (reportUnnecessaryIsInstance). Drop it and the now-unused
``AttrDict`` import.
- ``picopt/walk/scheduler.py``: ``_submit_ready_job`` was rank C in
radon. Split into three:
* ``_drop_cancelled_ready_job`` — counter cleanup for a leaf of a
cancelled subtree (UnpackJob/RepackJob skip; only leaf jobs
decrement the parent counter).
* ``_track_submitted_job`` — record an in-flight future under the
right map and update node state.
* ``_submit_ready_job`` — orchestrate: pop, cancelled-skip, submit,
track.
``_submit_ready_job`` is now rank A; the new helpers are A/B; behavior
unchanged.
Verified: ``make typecheck`` 0/0/0; ``make complexity`` no findings;
``uv run ruff check picopt/`` clean; ``uv run pytest`` 150 passed,
6 skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.