Batch mutations into one snapshot, and a DocxDiff-scored puzzle eval for the agent surface - #475
Open
JSv4 wants to merge 2 commits into
Open
Batch mutations into one snapshot, and a DocxDiff-scored puzzle eval for the agent surface#475JSv4 wants to merge 2 commits into
JSv4 wants to merge 2 commits into
Conversation
Every DocxSession mutation records a pre-op snapshot, and a snapshot deep clones every projected part — so its cost scales with the DOCUMENT, not with the edit. Measured on TestFiles/NVCA-Model-COI.docx (144 KB on disk), one snapshot retains ~7.5 MB, so a forty-edit sequence paid forty of those AND consumed forty entries of a twenty-deep ring. The sequence a caller had just applied was therefore already only half reversible by the time it finished. Both costs are properties of the loop, not of the work: N edits that form one intent deserve one snapshot and one undo step. Adds Batch(steps, options) for callers who can express steps as Func<EditResult> closures, and BeginBatch/EndBatch for those who cannot — a JSON dispatcher running its own switch, which is exactly the MCP case. Batch is implemented on top of the pair, so there is one mechanism. The 47 `_history.RecordPreOp(TakeSnapshot())` sites now route through RecordPreOpSnapshot(), which suppresses inside a batch. The suppression has to live at the call site rather than in UndoRing: TakeSnapshot() is evaluated before the ring ever sees the entry, so dropping it inside the ring would still pay for the clone. Failure policy cannot be used to escape the guarantee PR #459 established. Atomic (default) reverses on the first failure. Best-effort tolerates only failures that provably did not touch the document; a step that threw partway, or one the validator rejected after it had already written, reverses the whole batch regardless — inside a batch there is no per-step snapshot left to unpick that step alone. The two raw-op validation paths that hand-rolled their own pop-and-restore now use RollbackFailedOp() too, which removes the last non-uniform history call site. docxodus_mutations becomes genuinely atomic on the back of this, with no wire schema change. Its preview mode restores one snapshot instead of issuing N Undo() calls with N tracked by hand — that loop under-reverted whenever a step consumed more than one ring entry, making "nothing is left changed" a promise it could not keep. 20 tests (DS430-DS443) cover undo grain, both failure policies, damage overriding policy, nesting, the scope form, and the retained-memory shape. Full suite: 3476 passed, 0 failed.
The arcade proved the addressing model — anchors survive save/reopen, and the reconciler stays incremental while a host mutates the session behind it. It proved nothing about the surface we ask agents to drive, because it drives raw.replaceXml: the escape hatch for callers who want to hand-author OOXML, which is the opposite of what the tool surface is for. A level is a pair of documents and a budget: transform start into target in at most `par` calls, using only the grouped tools and anchor addressing. A level is solved when DocxDiff between the player's document and the target returns zero revisions, so the scoring function is the comparison engine itself — exact, unarguable, and the differentiator no demo surface touches. Levels declare both documents as paragraph lists rather than shipping binaries: they diff in review, and one builder constructs both sides so a scoring difference can only come from the player's edits. Reference solutions address blocks by content through FindAllByText — the op behind docxodus_search — so a reference cannot accidentally be easier than the task by starting with ids nobody has yet. CI keeps the LEVELS honest rather than testing the session: the reference solves the level (PE001), does so within par (PE002), and the start does not already score as solved (PE003). PE003 is the load-bearing one — a mis-built target would otherwise let an empty solution pass every level and report the pack at 100%. PE004 pins that the scorer rejects a partial solve; PE005 applies a whole level as one batch, since a plan is exactly what batching exists for. L01-clause-order ships first, and already shows the surface something: a two-paragraph clause costs two MoveBlock calls because there is no move-the-section op, which is the kind of affordance gap this pack exists to surface. Also corrects the arcade's og:/twitter: copy, which read as a live WAD parser where wad2cart.mjs rasterizes E1M1 at build time. freedoom-e1m1.js was already accurate; only the social meta overstated it. Full suite: 3481 passed, 0 failed. Library 115 warnings, tests 618 — both unchanged from baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, from an assessment of what the arcade actually proves.
The arcade proves the addressing model: anchors survive save/reopen, and the reconciler stays incremental while a host mutates the session behind it. It proves nothing about the surface we ask agents to drive, because it drives
raw.replaceXml— the escape hatch for callers who want to hand-author OOXML. That gap is what these two changes close from opposite ends.1.
DocxSession.Batch— one snapshot, one undo stepEvery mutation records a pre-op snapshot, and a snapshot deep-clones every projected part, so its cost scales with the document, not the edit. Measured on
TestFiles/NVCA-Model-COI.docx(144 KB on disk), one snapshot retains ≈7.5 MB — so a forty-edit sequence paid forty of those and consumed forty entries of a twenty-deep ring. The sequence a caller had just applied was already only half reversible by the time it finished. Both costs are properties of the loop, not of the work. Agent callers hit it hardest, because a plan is naturally a list of edits.Batch(steps, options)for callers who can express steps asFunc<EditResult>closures;BeginBatch/EndBatchfor those who cannot — a JSON dispatcher running its own switch, which is exactly the MCP case.Batchis implemented on the pair, so there is one mechanism._history.RecordPreOp(TakeSnapshot())sites now route throughRecordPreOpSnapshot(), which suppresses inside a batch. The suppression has to live at the call site, not inUndoRing—TakeSnapshot()is evaluated before the ring ever sees the entry, so dropping it inside the ring would still pay for the clone.RollbackFailedOp(), removing the last non-uniform history call site.docxodus_mutationsbecomes genuinely atomic on the back of this, with no wire schema change. Itspreviewmode now restores one snapshot instead of issuing NUndo()calls with N tracked by hand — that loop under-reverted whenever a step consumed more than one ring entry, making "nothing is left changed" a promise it could not keep. A step that mutated before failing now reportsrolledBack: truewitheditsApplied: 0rather than describing a half-applied document as a partial success.20 tests (DS430–DS443) cover undo grain, both failure policies, damage overriding policy, nesting, the scope form, and the retained-memory shape.
2. The puzzle eval — measuring the surface agents actually get
A level is a pair of documents and a budget: transform
startintotargetin at mostparcalls. A level is solved whenDocxDiffreturns zero revisions against the target, so the scoring function is the comparison engine itself — exact, unarguable, and the differentiator no demo surface touches.FindAllByText— the op behinddocxodus_search— so a reference cannot accidentally be easier than the task by starting with ids nobody has yet.L01-clause-orderships first and already shows the surface something: a two-paragraph clause costs twoMoveBlockcalls because there is no move-the-section op. That is the kind of affordance gap the pack exists to surface.Also
The arcade's
og:/twitter:copy read as a live WAD parser, wheretools/wad2cart.mjsrasterizes E1M1's geometry at build time and ships static data.freedoom-e1m1.jswas already accurate; only the social meta overstated it.Not done here
The batch primitive is core + facade + MCP only. The WASM bridge, npm/TS, and the stdio host + Python client do not yet expose it, so the ripple checklist is deliberately incomplete — flagging rather than hiding it. The MCP surface was the one that mattered for the eval, and it needed no new wire schema.
Testing
Generated by Claude Code