Skip to content

repsel: rule-1 provenance for unbound allocations — the 506 are 96% record literals, and Perry's own CJS IIFE is the wall (#7152 follow-up) #7170

Description

@proggeramlug

Follow-up to #7152, which measured that 506 of 746 Ptr<Shape> candidates in
dependency JS die at rule 1 (allocations never bound to a local) and concluded
"the dep-JS lever is rule 1, and its fix is anchoring the proof to an allocation
site rather than to a binding."

I re-measured on a second, independent slice of the same corpus and then
classified the population by mechanism rather than by position. The headline
number replicates. The reading of it does not survive, in three places, and the
correction changes what should be built first.

Short version: the dependency-JS wall is not a missing soundness argument. It
is Perry's own CommonJS wrapper — for the third time.
#7139 found the wrap
preamble arming the rule-5 barrier in 100% of CJS modules; #7152 found __cjs_module
accounting for 100% of the rule-2 bare references; this issue finds that the
wrap's IIFE turns every module-level function into a closure, which makes
#7107's return-shape provenance structurally unreachable in CommonJS. Same
self-inflicted class, same kind of fix: recognition, not relaxation.


1. Method, and what in it is a proxy

Compiler arm. wt-7148 @ 6a3b5bff8 — the #7149 branch tip, tracked tree
clean, perry-dev binary. Caveat, stated because CLAUDE.md requires it: the
binary's mtime is ~90 s before the commit timestamp (built, then committed),
and I reused it rather than rebuilding — local disk fell from 26 GB to 12 GB free
during the session from other agents' builds. The tracked tree is clean at that
commit and the element rule is present in the binary; the risk is low for a
static census, and it is nil for the code-derived findings in §5–§6, which are
proofs from source, not samples.

Corpus. real-apps/scriptc/node_modules, every third __esModule .js
file by sorted path — 197 selected of 589, 196 producing a report.
perry compile <m>.js --opt-report=json --no-link, each module copied to its own
directory so it compiles natively rather than routing to the removed V8 fallback.

This is not #7152's exact 180-module selection, and I could not reconstruct
it. Treat the two as two samples of one corpus. They agree on the headline share
and disagree on the position mix — see §3.

Two measurement points, and they are not the same population. The
--opt-report buckets are measured on HIR after state_desugar,
deforestation, inline_functions, loop unrolling and the async/generator
transforms. My mechanism census is measured on source AST. Between them sit
transforms that move allocations across positions — inlining in particular turns
call arguments into bindings. Where the two disagree I say so and I trust the
compiler-side number for anything about what the analysis sees, and the
source-side number for anything about what dependency code actually contains.
Neither is a claim about any other corpus.

Nothing was timed. No runtime numbers appear below; the host was building for
other agents throughout. All evidence is static counts, emitted-IR facts, and
code.


2. Compiler census — #7152 replicates

196 modules, 1352 Ptr<Shape> candidates:

bucket count share
rule 1 — allocated in expression position, never bound 963 71.2%
rule 5 — module barrier still armed 219 16.2%
rule 2 — containment 149 11.0%
class admission 7 0.5%

Outcomes across the whole corpus: denied 1338, selected 3, consumed 11.
Only 5 of 196 modules (3%) get any promotion at all. 58 of 196 (30%) still
carry an armed rule-5 barrier after #7139. Of the 149 rule-2 denials, 140 are
__cjs_module
#7152's finding, replicated.

Rule 1 at 71.2% vs #7152's 67.8% — the headline is solid.

Three facts #7152 did not report, all from the same records:

  • 95.6% of unbound allocations are object literals, not new C(...)
    921 of 963. The 42 real constructors are led by new Promise (23), which is a
    native base and would fail class admission even with perfect provenance. This
    population is record literals, not class instances.
  • 91.6% sit in closure regions — 882 of 963 (method 46, function 34,
    module-init 1). §6 explains why, and it is the whole story.
  • 92.2% are at loop depth 0 and only 20 of 963 are in a per-element
    callback. By the report's own hotness proxy this population is cold.

3. The mechanism distribution (the deliverable)

Source-AST census over the same 197 modules, counting allocations that are not
the direct initializer of a variable declaration
— i.e. exactly what
ptr_shape_report.rs::unbound_new_sites looks for — restricted to closed-shape
object literals, the 95.6% that Perry lowers to Expr::New{__AnonShape_N}.
n = 1651.

true syntactic consumer count share
call argument 538 32.6%
  → Object.defineProperty descriptor 265 16.1%
  → bare-identifier callee 149 9.0%
  → method call 85 5.1%
  → IIFE / super / Object.freeze / assign 39 2.4%
nested inside another object literal 404 24.5%
array element 167 10.1%
return 146 8.8%
assign to an identifier (reassignment) 100 6.1%
store to a property 96 5.8%
conditional branch 61 3.7%
default parameter value 55 3.3%
binary operand 39 2.4%
constructor argument (new C({...})) 33 2.0%
yield / class-field init / member access 12 0.7%

Of the 96 property stores, 44 target exports, 12 target this, 4 target
module; the rest are scattered. Of the 538 call arguments, 265 are
defineProperty descriptors
— the __esModule marker and the re-export
forwarders, one or more per transpiled module.

Grouped by what a fix would have to be:

tier mechanism count share
T1 the object becomes a component of another object (nested literal, array element, property store) — heap-resident by construction 667 40.4%
T2 consumed by an opaque runtime callee (Object.defineProperty/freeze/assign) 279 16.9%
T3 flows into a same-program callee — parameter-shape territory 292 17.7%
T4 returned to a caller — return-shape territory 146 8.8%
T5 bound, but by a shape rule 1 rejects (reassignment 100, conditional join 61) 161 9.8%
T6 other (default parameter, binary operand, yield, …) 106 6.4%

Sampling caveat. The top 5 modules contribute 42.5% of all sites (median
per module: 3). This is a bundle-heavy corpus — TypeScript, Vite chunks, ESLint.
The distribution is a proxy for "transpiled CJS dependency code", not for JS
generally.


4. The leads in the brief, answered from the data

  • require() results — zero. Not one unbound literal is bound from a
    require() call in this corpus; require appears as a callee of literal
    arguments, never as a producer of one.
  • conditionals — 61 of 1651 (3.7%). Perry's own initializer bucket is
    47 of 963 (4.9%). Real, small, not the lever.
  • property gets — do not appear as a producer of these allocations at all,
    and §5.3 shows seeding from them is actively dangerous.
  • method calls — 85 of 1651 (5.1%) as consumers of an unbound literal; as
    producers they are the residue perf(repsel): return-shape facts — Ptr<Shape> survives the return escape (#7034 §4) #7107 cannot reach (§6).

5. Three corrections to #7152's reading

5.1 "constructor argument 182" is not new Outer(new Inner()) — it is nested object literals

A closed-shape literal lowers to new __AnonShape_N(v0, v1, …) whose
constructor arguments are the property values

(perry-hir/src/lower/expr_object.rs:632, perry-hir/src/dynamic_import.rs:1615).
ptr_shape_report.rs::scan_expr then labels every argument of any Expr::New
"constructor argument". So {a: {b: 1}} files the inner literal under
constructor argument.

At source level, genuine new C({...}) arguments are 33 of 1651 (2.0%) while
nested literals are 404 (24.5%). #7152's second-largest bucket is
overwhelmingly the nested-literal case.

This matters because a nested literal is not an independent promotion
opportunity
. It is a field value of a parent allocation that is itself unbound.
Proving its shape licenses nothing on its own.

5.2 The return bucket is already served on the producer side — the denial over-counts

report::deny_alloc_site is called unconditionally at the top of
collect_shape_proven_ptr_locals
(ptr_shape.rs:311), before any seeding runs,
and again on every early-bail path (report_early_bail, :274). It has no
knowledge of what the later passes do.

Meanwhile ptr_shape_returns.rs:169-188 already admits a bare
return new C(...) / return {…} with no local at all
as a return-shape
producer.

So a return {a, b} site is reported as a rule-1 denial while simultaneously
feeding an existing mechanism
. The 506 / 963 is a count of syntactic sites,
not of missed opportunities
, and it is inflated by at least the served part of
the return bucket plus the entire T1/T2 population that no provenance rule could
ever want. Any scheduling decision taken off that number is taken off a number
that is not measuring what it appears to.

5.3 Widening the seed to PropertyGet would silently break #7139

#7139's exemption is sound on two legs, and the first is:

The target is not a candidate. … The third clause admits only initializers
that are not fresh allocations at all — a field read, a function value, or
nothing — so an exempted target can never be promoted, and, being a
whitelist, it stays true if the seed set widens
.
collectors/cjs_scaffolding.rs:47-55

init_is_never_a_seed's whitelist is exactly PropertyGet / Closure /
Undefined / absent. And the wrap template
(compile/cjs_wrap/wrap.rs:758-760) binds

const __cjs_module = { exports: {} };
var module  = __cjs_module;
var exports = __cjs_module.exports;   // ← PropertyGet

So exports is bound by a PropertyGet, and
Object.defineProperty(exports, "__esModule", …) is the site #7139 exempts from
arming the module barrier. Seed rule 1 from PropertyGet and exports becomes a
promotable candidate whose mutator no longer arms the barrier — the first leg is
gone, and only rule-2 containment (the "belt" that rule 5 is the "braces" for)
stands between that and an unsound promotion.

Binding constraint for every increment below: seeding from PropertyGet or
Closure requires re-deriving or reverting #7139 in the same PR.
Seeding from
Call (#7107's existing widening) and from New in expression position do
not touch the whitelist, because neither is in it.


6. ★ The decisive experiment: Perry's own CJS IIFE is the wall

Same source body, compiled three ways, --opt-report=json:

probe wrapper result for const p = mk(i)
p7_esm_store.ts none region functionselected, and consumed via ptr_shape_get_number
p8_iife.ts (function(){ … })() region closurenot even a candidate
p1_direct.js CJS (exports. + defineProperty) region closurenot even a candidate
p6_cjs_nodp.js CJS, no defineProperty region closurenot even a candidate
function mk(i) { return { a: i, b: i + 1 }; }          // producer: bare `return {…}`
function run(n) {
  let s = 0;
  for (let i = 0; i < n; i++) { const p = mk(i); p.a = p.a + 1; s += p.a + p.b; }
  return s;
}

p8 differs from p7 only by the IIFE. p6 differs from p1 only by the
defineProperty, and behaves identically — so this is not the rule-5 barrier
and not #7139's territory. It is the wrapper.

Mechanism. cjs_wrap emits every CommonJS module body inside an IIFE
(wrap.rs:889, {cjs_preamble}\n\n{body_for_iife}, "reproduces the historical
IIFE text byte-for-byte"). Inside it, mk is a closure, not an entry in
hir.functions. #7107's consumer is
Stmt::Let { init: Expr::Call { callee: Expr::FuncRef(id) } }
(ptr_shape_returns.rs:348-369) — a bare Expr::FuncRef and nothing else. A
closure callee is not a FuncRef, so the seed never fires.

That is why 91.6% of dependency-JS alloc sites are in closure regions. It is
one cause, it is Perry's own, and it makes the entire return-shape mechanism —
the one mechanism that already handles unbound allocations — unreachable across
the CommonJS ecosystem.

6.1 …and even where it fires, scalar replacement usually wins

p4/p5 (ESM, same code without the in-loop field store) report
selected and then unconsumed — scalar_replaced: escape_news deleted the
object outright, so the Ptr<Shape> proof emitted nothing. Only p7, which adds
one in-loop field store, is consumed.

This is mechanism 3 from the plan's "a promotion goes unconsumed three ways",
measured on exactly this idiom. The benefit of any return-bucket work must be
measured on shapes that survive scalar replacement
, or it will report promotions
that emit no bytes.


7. Composition — settled, not deferred

Scalar replacement (escape_news.rs, #7115): cannot collide at the seed.
All three of its candidate finders match only a Stmt::Let-bound allocation
(escape_check.rs:23-29, escape_objects.rs:219-252, escape_arrays.rs:351-380).
Expression-position allocations — the entire 963 — are never scalar-replacement
candidates. The collision surface is downstream: if an increment produces a
bound local (as #7107's does at the caller), it lands in scalar replacement's
territory, and let_stmt.rs checks ctx.scalar_replaced first, in file order,
at both the Let (:773-828) and every PropertyGet (property_get.rs:715-804,
comment at :786). The existing tie-break already logs the loser
(note_ptr_shape_scalar_replaced, let_stmt.rs:1973-1994). Requirement for any
new rule: consult non_escaping_news, or be consumed at a site that checks
ctx.scalar_replaced first — otherwise it reproduces the known dead-fact pattern.

Deforestation (#7150): no interaction with object provenance, at all.
Its seed is hard-coded to Expr::Array(elems) with elems.is_empty()
(deforest/detect.rs:206-208); there is no Expr::New/object variant anywhere in
detect.rs, producer_rewrite.rs or call_sites.rs. It is also an HIR pass
run from collect_modules.rs:1914, finishing for every module before
compile_module (run_pipeline.rs:4305) runs any collector. #7150 is a real
conflict with #7149's array rule (E1/E2) and is not a constraint on object
provenance. This closes the question the brief asked me to answer up front: for
the work below, there is nothing to compose with.

#7139's whitelist: §5.3. Call and New seeds are safe; PropertyGet and
Closure seeds are not.

GC contract (plan + #6990), and it is the same for every increment below.
None of them introduces a new pointer position. Every promoted value stays a
NaN-boxed slot bound by js_shadow_slot_bind; the raw pointer is region-local SSA
re-derived from that bound slot at each access; TaPtr's callee-side no-bind
shortcut is never copied (GC_TYPE_OBJECT is movable); write barriers on stores
are untouched. Increments R1/R2 promote a caller-side local that already
exists
and already has a slot — they change which lowering reads it, not how it
is rooted. R3/R4 would introduce values with no binding, and that is where the
contract needs new argument; it is one of the reasons they rank last.


8. Ranked increments

R0 — instrument: stop reporting already-served and structurally-dead sites as rule-1 denials

Report-only; no emitted byte changes.

  • What. Suppress the alloc-site denial when the site is (a) in return
    position of a function that carries a return-shape fact — already served; or
    (b) a property value of another unbound allocation — a component, never an
    independent candidate. Optionally split the "constructor argument" label into
    nested object literal vs new C(arg), which today are merged (§5.1).
  • Admits. Nothing. Removes ≈ the served part of 201 return + the nested share
    of 197 constructor-argument denials from a number that is currently used to
    schedule work.
  • Soundness. N/A — the collector's returned facts are untouched; recording
    happens next to the continue, never instead of it.
  • GC. Unchanged.
  • Composition. None.
  • Measurement. Rule-1 count falls on the dependency corpus; repsel-census
    fixture counters unchanged; a byte-identical-object A/B proves it is report-only.

Do this first. §5.1 and §5.2 are both cases of a number meaning something
other than what it looks like, and #7152 and this issue were both scheduled off
that number. This is CLAUDE.md failure mode 4 in the instrument itself.

R1 — ★ make #7107 reachable inside the CJS wrapper

  • What. Two options; (b) is the smaller change.
    (a) Have cjs_wrap lower the module body so that top-level
    function declarations remain hir.functions entries rather than closures.
    (b) Extend perf(repsel): return-shape facts — Ptr<Shape> survives the return escape (#7034 §4) #7107's consumer to accept a callee that is LocalGet(id)
    where id is bound by exactly one Stmt::Let { init: Expr::Closure },
    never reassigned and never written through a capture — and the corresponding
    producer-side extension so such a closure is analysed as a producer.
  • Soundness sketch. The consumer's requirement is "the callee names one
    statically-known function"
    (ptr_shape_returns.rs:360-362: "Anything computed
    could be rebound"). A local bound by exactly one Let to a closure literal and
    never reassigned names one statically-known function by the same argument that
    Expr::FuncRef does — the single-Let condition is the identical test rule 1
    already applies to object provenance. Everything else in perf(repsel): return-shape facts — Ptr<Shape> survives the return escape (#7034 §4) #7107 is unchanged:
    the producer proof, the class agreement across returns, the module-barrier kill.
  • Admits. The return bucket, end-to-end, for CommonJS. Source-level: 146
    return-position literals, of which 91 are in a named function decl/expr and
    43 already have ≥1 same-module const x = f(…) call site — that 43 is the
    count that would promote today if the wrapper were not in the way. It also
    unblocks perf(repsel): return-shape facts — Ptr<Shape> survives the return escape (#7034 §4) #7107 for the ~98.8% of dependency modules perf(repsel): exempt CommonJS module scaffolding from the Ptr<Shape> rule-5 barrier #7139 made rule-5-eligible,
    whose regions are 91.6% closures.
  • GC. No new pointer position; the caller's local is an ordinary bound slot
    (§7).
  • Composition. Does not widen the seed to PropertyGet/Closure — the
    seed stays Let { init: Call }, so perf(repsel): exempt CommonJS module scaffolding from the Ptr<Shape> rule-5 barrier #7139's whitelist is untouched (§5.3). It
    does land in scalar replacement's territory at the caller, and §6.1 measured
    that scalar replacement wins on the plain idiom: the acceptance fixture must
    carry a field store, or the PR will report promotions that emit nothing.
  • Measurement. Opaque js_* call count in a CJS-wrapped fixture, before
    and after, with call sites checked in the emitted IR — not an --opt-report
    counter (that is exactly what §6.1 shows lying). Plus dependency-corpus
    consumed, today 11 across 196 modules.

R2 — return-shape consumers beyond the direct call

  • What. Method-call producers (obj.m()) and cross-module producers.
  • Soundness. Method calls need dispatch stability, which rule 4 already has
    machinery for. Cross-module is structurally blocked, not merely
    unimplemented
    : imported callees lower to Expr::ExternFuncRef { name },
    which carries no FuncId, so return_shape_functions's HashMap<u32, String>
    cannot be keyed for them. It needs a name-keyed fact table plus a module
    ordering guarantee.
  • Admits. 85 method-call consumers at source level (5.1%); cross-module is
    unmeasured (§10).
  • Rank. After R1, and only if R1's measurement shows the return bucket pays.

R3 — parameter-shape facts (all-call-sites-agree)

R4 — heap-position provenance (T1) — do not attempt incrementally

T1 is 667 sites, 40.4% — the largest tier by a wide margin. Every one is "the
allocation becomes a field or element of another object". Containment is defined
as the object is unreachable from anywhere except this local; T1 is the negation
of that premise. No relaxation of rule 2 reaches it. This is what a self-describing
header and a shape tree buy, and it is the substance of §9.

R5 — the #7142 fallback tier

Where a static proof cannot reach — T3's non-agreeing call sites and any T1 access
— an inline keys-token guard is the right tier: it pays where the receiver has no
static class, which is precisely the T1/T3 shape. It is a runtime check, so it
composes with none of the above and needs no provenance at all. Sequence it after
#7142 lands and reports its own prize.


9. Does this change the #6759 calculus? — Yes, decisively, for dependency JS

The standing fork is "containment relaxation vs V8-model rebuild", and the plan
says each opened position tells you how much that escape shape was costing. This
distribution is the answer for dependency code:

  • 40.4% (T1) is allocations that become part of another object. Containment
    cannot reach these by construction — not "not yet", but by the definition of
    the proof.
  • 17.7% (T3) needs interprocedural agreement across all call sites.
  • 16.9% (T2) is defineProperty descriptors — never promotable and never
    worth promoting.
  • 8.8% (T4) is the return bucket, and §5.2/§6 show it is already served by an
    existing mechanism that Perry's own wrapper disables.

So: containment relaxation is finished as a dependency-JS lever. #7149 opened
the array-element position and got 1 of 746 there; #7152 showed call-argument
is 5 of 746; this issue shows the remaining tractable position (return) is not a
missing proof at all. After R0 and R1 there is no third containment position worth
opening for this population.

That is not an argument to start #6759 now. The honest reading is:

  1. The cheap wins left in dependency JS are all Perry's own scaffolding
    perf(repsel): exempt CommonJS module scaffolding from the Ptr<Shape> rule-5 barrier #7139 (done), repsel: dependency JS is walled by rule 1 (unbound allocations), not containment — 506 of 746 candidates #7152's __cjs_module (in flight), and R1's IIFE. None of them is
    a soundness relaxation; all are recognition. Finish them, because they are
    nearly free and because each currently makes the corpus look harder than it is.
  2. Then the fork is real, and it is between R5 (repsel Phase 5a: route the class-id dispatch tower to __pshape under an inline keys check (batch.ts: 4 guard calls + 3 by-name field calls per call) #7142's runtime tier) and
    Architecture: adopt V8's object-model construction — explicit runtime state, self-describing headers, shape tree (phases A–C) #6759
    — not between Architecture: adopt V8's object-model construction — explicit runtime state, self-describing headers, shape tree (phases A–C) #6759 and more containment positions. R5 is the cheaper
    probe of the same hypothesis ("make escaping not matter"), and it produces a
    measured prize on the T1/T3 population before anyone commits to the rebuild.
  3. The app-TS population is different and this does not speak for it. perf(repsel): open the array-element escape for Ptr<Shape> (#7034 §3) #7149
    pays on build-then-consume TypeScript. The fork should be decided per
    population, and a decision to rebuild should not be justified with these numbers
    alone.

The thing to avoid, per the plan, is drifting into #6759 after five incremental
PRs. This distribution says: two more incremental PRs (R0, R1), then measure R5,
then decide.


10. What I could not measure, plainly

  • Anything about speed. No benchmark was run; the host was building for other
    agents throughout and disk fell to 12 GB free. Every number here is a static
    count, an emitted-IR fact, or a source classification.
  • Scalar-replacement survival rate over the 146 return sites. §6.1 shows it
    wins on the plain idiom and loses when a field store is added, but the corpus
    split needs a compiler-side counter that does not exist. This is the single
    most important unknown for R1's benefit story
    and should be the first thing
    R1's PR instruments.
  • Whether the source-AST distribution survives the HIR transforms. My faithful
    emulation of unbound_new_sites over source gives array-element 167 where the
    compiler reports 14, and call-argument 562 where it reports 190. Inlining,
    desugaring and the generator transforms run in between. §3 is a statement about
    what dependency code contains, not about what the collector sees.
  • repsel: dependency JS is walled by rule 1 (unbound allocations), not containment — 506 of 746 candidates #7152's exact 180-module selection. Not reconstructable from the issue text;
    §2's slice is a different sample. Headline share agrees (71.2% vs 67.8%),
    position mix does not (my return share 20.9% vs 12.6%; constructor-argument
    20.5% vs 36%) — consistent with §5.1, since the constructor-argument bucket is
    really a nested-literal bucket and nesting depth varies by bundle.
  • Cross-module return-shape reach. R2's cross-module half is blocked
    structurally; I did not measure how many producers live in a different module
    from their consumers.
  • x86-64. Everything here is macOS arm64.
  • Whether hir.functions contains closures, which decides whether R1(b) needs
    a producer-side change as well as a consumer-side one. The code did not settle
    it and I did not trace closure call-site lowering.

Reproduction

Census: perry compile <module>.js --opt-report=json --no-link on every third
__esModule .js in real-apps/scriptc/node_modules (197 of 589), each copied
to its own directory. The report goes to stderr, not stdout. Mechanism census:
TypeScript 5.9.3's parser, emulating ptr_shape_report.rs::scan_stmts/scan_expr
context propagation, counting allocations that are not a variable declaration's
direct initializer. Probes p1p8 in §6 are 8–10 lines each and are the
load-bearing evidence; they need no corpus.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions