Skip to content

Add verbose modes. Add 1.11 support back. Various fixes. Add color#37

Open
IanButterworth wants to merge 9 commits into
mainfrom
ib/verbose
Open

Add verbose modes. Add 1.11 support back. Various fixes. Add color#37
IanButterworth wants to merge 9 commits into
mainfrom
ib/verbose

Conversation

@IanButterworth
Copy link
Copy Markdown
Collaborator

@IanButterworth IanButterworth commented Jul 9, 2025

This became a bit of a Claude take the wheel and fix everything...

Closes #41


Add verbose modes (and broader Julia compatibility)

Adds opt-in verbose output to info_cachefile for inspecting pkgimage
contents in detail, refreshes compatibility with current Julia versions
(1.11–nightly), and cleans up the surrounding code.

Written with the assistance of generative AI (GitHub Copilot, Claude
Opus 4.7).

New verbose keyword on info_cachefile

info_cachefile(pkg; verbose=:none) now controls how much detail is
printed when a PkgCacheInfo is shown:

value output
:none summary counts only (current default; unchanged behavior)
:internal also list internal methods grouped by function, with their specializations
:external also list methods this package adds to functions defined elsewhere, plus any new external specializations
:all union of :internal and :external

Method, specialization, and external-method lists are sorted for stable
output. External entries are filtered to those whose defining module is
not part of the package being inspected, so the section actually shows
extension of upstream functions rather than re-listing the package's
own methods.

Internal-method counting is reworked to walk Core.methodtable (renamed
from Core.GlobalMethods in 1.12) with a fallback to
info.internal_methods for older versions where the API isn't public.

Verbose rendering is colorized: headers, counts, module names, and the
function-name groupings are printed via printstyled so the output
highlights structure in color-capable terminals while remaining a no-op
under sprint and other colorless IOs. Method objects are rendered
through their own color-aware show (preserving source-location
highlighting), and the caller's IOContext is propagated through
Base.show_tuple_as_call so specialization signatures pick up color
too.

Julia 1.11 / 1.12 / 1.13 / nightly compatibility

The cache-file plumbing has shifted shape across recent Julia versions;
this PR teaches the inspector to handle each:

  • 1.11 (8-element restored sv: (modules, init_order, extext_methods, new_ext_cis, method_roots_list, ext_targets, edges, cachesizes)), 1.12/1.13, and master layouts are all unpacked
    by a single _unpack_restored_sv helper.
  • parse_cache_header's return tuple grew a syntax_version element on
    master and replaced prefs/prefs_hash with prefs_blob, shifting
    clone_targets from index 7 to index 6. The PR locates
    clone_targets by type (Vector{UInt8}) so the code works on 1.12,
    1.13, and nightly without version branching.
  • Core.GlobalMethods / Core.methodtable access is guarded so 1.11
    (which exposes neither publicly) falls through to
    info.internal_methods instead of erroring at top level.
  • ocachefile_from_cachefile is only consulted when clone_targets is
    non-empty, matching upstream behavior when no pkgimage .dylib is
    associated with a .ji.
  • Package lookup in info_cachefile(pkg::PkgId, …) uses
    Base.locate_package_load_spec when available so the cache's
    recorded syntax version is honored. This fixes spurious "all cache
    files are stale"
    errors on Julia 1.13+ when the active manifest was
    resolved with a different Julia version than the running one.

CI matrix expanded to '1.11', '1.12', '1.13-nightly', 'nightly'.
Project.toml is now a workspace with test/ and docs/ members
(adds test/Project.toml), and the julia compat is lowered to
"1.11".

Repeat-call safety

Loading the same pkgimage twice in one session via
jl_restore_package_image_from_file / jl_restore_incremental
corrupts the runtime and segfaults. info_cachefile now caches results
in a module-local _INFO_CACHE keyed by the realpath of the cache
file; on a repeat call it emits a warning and returns the prior
PkgCacheInfo (re-wrapped with the freshly requested verbose
setting) instead of re-invoking the loader. The fast-path runs before
any cache-file open, CRC check, or _tryrequire_from_serialized call.

Code-quality cleanup

Folded into the same PR while the file was open:

  • Tighter types. PkgCacheInfo fields move from Vector{Any} to
    concrete element types (Vector{Method} for external_methods and
    internal_methods, Vector{Core.CodeInstance} for
    internal_method_specializations and new_specializations). The
    fields were already populated with those concrete types; the loose
    typing only cost inference quality.
  • Kwarg consistency. Every public info_cachefile entry point now
    accepts verbose as a keyword argument, matching the docstring.
  • Defensive helpers. count_module_specializations and the verbose
    renderers gracefully skip non-Method-backed CodeInstances.
  • Real errors. return ArgumentError(...) on the invalid-header /
    bad-CRC paths is now throw(ArgumentError(...)) so callers actually
    see a failure.
  • Dedupe by Method, not binding name. show_verbose_internal_methods
    used to iterate names(mod; all=true) and call methods(obj) per
    binding, so gensym aliases (e.g. var"#15#val" aliasing
    normalize_hue) duplicated whole sections. It now collects into a
    Method-keyed set first and groups under the true function-owner
    symbol, so each Method is listed exactly once.
  • Dead-code / docs / typos. Drops the unused assert_havelock
    import, the stub moduleof(::Module) method, redundant
    isa(_, Core.CodeInstance) chains, and a double blank line. Fixes
    auxillaryauxiliary and !!! warn!!! warning so the
    Documenter admonition renders. Sets checkdocs=:exports so internal
    helpers don't break the docs build.
  • Renames. Internal helpers _n / _mod / _hdr_count /
    _modname / _header for self-documenting call sites. The test-only
    zero-arg PkgCacheSizes() and two-arg
    PkgCacheInfo(cachefile, modules) constructors moved out of the
    source module into test/runtests.jl as empty_cachesizes() /
    empty_info(...). Pkg.precompile() in the test runner is scoped
    to "Colors" (the only inspected package).
  • .gitignore. Excludes per-version root Manifests and
    test/Manifest.toml.

Test plan

Pkg.test() passes on Julia 1.11, 1.12, 1.13.0-rc1, and
1.14.0-DEV (9/9 tests on every version). Verbose rendering was
spot-checked against the Pkg, REPL, Dates, and Colors pkgimages on
1.14.0-DEV.

Commits

  1. Add verbose modes and Julia 1.12/1.13/nightly compatibility
  2. Support Julia 1.11
  3. show_verbose_internal_methods: dedupe by Method, not binding name
  4. show: colorize output and use color-aware Method rendering
  5. info_cachefile: cache result to avoid double-load segfault
  6. cleanup: tighten types, kwarg-ify verbose, defensive helpers
  7. cleanup: prune dead code, tighten cache fast-path, fix typos
  8. cleanup: rename color helpers, move test-only ctors, scope precompile

@IanButterworth IanButterworth marked this pull request as draft July 9, 2025 17:58
Comment thread src/PkgCacheInspector.jl Outdated
@timholy
Copy link
Copy Markdown
Owner

timholy commented Jul 25, 2025

Do you think this might be better as a filter rather than a printer? It seems you can do more if you pass back the subset of methods rather than just printing them.

@IanButterworth
Copy link
Copy Markdown
Collaborator Author

I think there's uses for both. With "verbose" stuff it's kind of useful to just dump things into a text file etc.

Base automatically changed from ib/estimate_size to main July 29, 2025 00:44
Add `verbose` modes to `info_cachefile` for controlling how much detail is
printed when displaying a `PkgCacheInfo`:

  - `:none`     - summary counts only (default)
  - `:internal` - list internal methods and their specializations
  - `:external` - list external methods and new specializations
  - `:all`      - list everything

Method, specialization, and external-method lists are sorted for stable
output, and external entries are filtered to those whose defining module
is not part of the package being inspected.

Internal-method counting is reworked to use `Core.methodtable` (renamed
from `Core.GlobalMethods` in Julia 1.12), with a fallback for older
nightlies.

Package lookup in `info_cachefile(pkg::PkgId, ...)` now uses
`Base.locate_package_load_spec` when available so the cache's recorded
syntax version is honored; this fixes spurious "all cache files are
stale" errors on Julia 1.13+ when the active manifest was resolved with
a different Julia version than the running one.

`parse_cache_header`'s return tuple changed shape on master (a new
`syntax_version` element was appended and `prefs`/`prefs_hash` was
replaced with `prefs_blob`), shifting `clone_targets` from index 7 to
index 6. Locate `clone_targets` by type (`Vector{UInt8}`) so the code
works on 1.12, 1.13, and nightly without version branching.

Set up `Project.toml` as a workspace with `test` and `docs` members
(adding `test/Project.toml`), and expand CI to cover 1.12, 1.12-nightly,
1.13-nightly, and nightly.

Co-authored-by: GitHub Copilot <copilot@github.com>
Extend the verbose-modes PR to cover Julia 1.11 as well as 1.12, 1.13 and
nightly:

- `_unpack_restored_sv` learns the 8-element 1.11 layout
  `(modules, init_order, extext_methods, new_ext_cis, method_roots_list,
   ext_targets, edges, cachesizes)`.
- Guard `Core.GlobalMethods` / `Core.methodtable` access so 1.11 (which
  exposes neither publicly) falls through to the `info.internal_methods`
  source and otherwise reports no per-module counts rather than erroring
  at top level.
- Gate the `ocachefile_from_cachefile` fallback on a non-empty
  `clone_targets`, matching upstream behavior when no pkgimage `.dylib`
  is associated with the `.ji` cache.
- CI matrix: add `'1.11'`; keep `'1.12'`, `'1.13-nightly'`, `'nightly'`.
- Compat: lower `julia` to `"1.11"`.
- `docs/make.jl`: set `checkdocs=:exports` so internal helpers
  (`_unpack_restored_sv`, `_split_code_instances`, `show_verbose_*`) do
  not break the docs build.
- `.gitignore`: exclude per-version root Manifests and `test/Manifest.toml`.

Tested with `Pkg.test()` on 1.11, 1.12, 1.13.0-rc1 and 1.14.0-DEV; all 9
tests pass on every version.

Co-authored-by: GitHub Copilot (Claude Opus 4.7) <noreply@github.com>

Written with the assistance of generative AI (GitHub Copilot, Claude Opus 4.7).
IanButterworth and others added 6 commits May 20, 2026 22:32
`show_verbose_internal_methods` iterated `names(mod; all=true)` and called
`methods(obj)` for each binding. When multiple gensym-aliased bindings
referred to the same underlying function (e.g. `var"#15#val"` aliasing
`normalize_hue`), each alias produced its own section, listing the same
methods again under an unhelpful binding name. The summary count was
correct but the per-function breakdown could repeat methods many times.

Collect methods into a `Method`-keyed set first, group by the method's
function-owner symbol, then render. Each Method is now listed exactly
once, under its true function name.

Co-authored-by: GitHub Copilot (Claude Opus 4.7) <noreply@github.com>

Written with the assistance of generative AI (GitHub Copilot, Claude Opus 4.7).
Replace plain `print`/`println` of headers, counts, module names and
function-name groupings with `printstyled` calls so the rendered
`PkgCacheInfo` and `PkgCacheSizes` summaries highlight structure in
color-capable terminals. `printstyled` is a no-op for non-color IO, so
`sprint(show, info)` and other colorless contexts are unchanged.

Render `Method` objects via their own color-aware `show` (preserving
source-location highlighting) instead of `print`-stringifying them, and
propagate the caller's `IOContext` into `Base.show_tuple_as_call` so
specialization signatures pick up color too. Drop the buffered
`IOContext(method_buffer, stdout)` indirection in the external-methods
helper — it inherited `stdout`'s color regardless of the caller's IO.

Co-authored-by: Claude <noreply@anthropic.com>
Loading the same pkgimage twice in one session via
`jl_restore_package_image_from_file` / `jl_restore_incremental` corrupts
the runtime and segfaults. Track previously inspected cache files in a
module-local dict keyed by the realpath; on a repeat call, emit a warning
and return the prior `PkgCacheInfo` (rebuilt with the freshly requested
`verbose` setting) instead of re-invoking the loader.

Co-authored-by: Claude <noreply@anthropic.com>
Apply review cleanup cluster:

- Tighten `PkgCacheInfo` element types from `Vector{Any}` to `Vector{Method}` /
  `Vector{Core.CodeInstance}` for `external_methods`, `internal_methods`,
  `internal_method_specializations`, and `new_specializations`. These fields
  were already populated with the concrete types; the looser typing only cost
  inference quality.
- Switch the public `info_cachefile` entry points to accept `verbose` as a
  keyword argument so the docstring's promise (`; verbose=...`) is actually
  honored for every signature.
- Make `count_module_specializations` defensive against non-`Method`-backed
  `CodeInstance`s, matching the guards used elsewhere.
- Replace `return ArgumentError(...)` with `throw(ArgumentError(...))` on the
  invalid-header / bad-CRC paths so callers see a real error.
- Extract `_truncate_for` to remove the duplicated `Base.rtruncate` block.
- Fix `!!! warn` -> `!!! warning` so Documenter renders the admonition.
- Drop the dead `moduleof(::Module)` method.
- Rename a `methods` loop variable that shadowed `Base.methods`.
- Remove a stray double blank line.

Co-authored-by: Claude <noreply@anthropic.com>
- Drop unused `assert_havelock` / `Core.SimpleVector` imports.
- Fix `auxillary` → `auxiliary` and "gets returns by" typos in
  `PkgCacheSizes` docstrings.
- Collapse stale defensive `isa(_, Core.CodeInstance)` chains in the
  three verbose/show helpers now that the corresponding fields are
  typed `Vector{Core.CodeInstance}`. Drop the redundant
  `mod ∉ info.modules` re-filter in `show_verbose_external_methods`
  (`_split_code_instances` already partitions by module).
- Move the `_INFO_CACHE` lookup into the path-form
  `info_cachefile(pkg, path; verbose)` entry, keyed on the resolved
  load path. Repeat calls now short-circuit before opening the
  cachefile, validating CRC, or pulling deps via
  `_tryrequire_from_serialized`. Factored out `_with_verbose` /
  `_cache_key` helpers; the inner 6-arg method keeps a defensive
  lookup as a safety net.

Co-authored-by: Claude <noreply@anthropic.com>
- Rename `_n`/`_mod`/`_hdr` → `_count`/`_modname`/`_header` for
  self-documenting call sites.
- Move the test-only zero-arg `PkgCacheSizes()` and two-arg
  `PkgCacheInfo(cachefile, modules)` constructors out of the source
  module and into `test/runtests.jl` as `empty_cachesizes()` /
  `empty_info(...)` test helpers.
- Scope `Pkg.precompile()` in the test runner to `"Colors"` (the only
  package actually inspected by the tests).

Co-authored-by: Claude <noreply@anthropic.com>
@IanButterworth IanButterworth changed the title Add verbose modes Add verbose modes. Add 1.11 support back. Various fixes. Add color May 21, 2026
@IanButterworth IanButterworth marked this pull request as ready for review May 21, 2026 03:57
On Julia post-#58131 (single global jl_method_table), info.external_methods
contains *all* worklist-defined methods, not only those extending externally-
owned functions. The summary line previously printed "N external methods" with
the same count as "N internal methods", which was confusing and redundant.

Changes:
- Replace the "N external methods" summary line with "N methods extending
  external functions", computed via the new extending_external_methods helper,
  so the count reflects only the true external-extension subset.
- Fix show_verbose_external_methods to filter through extending_external_methods
  instead of iterating all of info.external_methods.
- Add an unexported extending_external_methods(info) helper. It filters
  info.external_methods to methods whose function-type owning module is outside
  the pkgimage's worklist modules.
- Rewrite the docstrings for external_methods and internal_methods fields to
  accurately describe their post-#58131 semantics (both contain all worklist
  methods; they differ only in object form and load-time role).
- Add four tests for extending_external_methods.
@IanButterworth
Copy link
Copy Markdown
Collaborator Author

@timholy I brought back 1.11 support because of #41 and because Claude makes it easier to maintain this kind of thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants