Cross-demo test gaps: phone routing, NF-e payload fidelity, output-message quality

[dangazineu]

Summary

Three assertion gaps that every example with a WhatsApp or fiscal leg shares today. Each one would let a real regression ship green. Filing as one issue because they're the same pattern (under-asserted wire-contract fields on a tool call we already capture) and the fix touches the same set of skeleton and live test files.

Reference bar: #34 set the assertion density we want skeleton tests to mirror — every wire field that doesn't depend on LLM phrasing should be pinned.

Gap 1 — Phone routing is unasserted

Every WhatsApp-touching test checks the message body for keywords but never asserts the phone field on z-api__send_text is the right number. The agent could send the buyer's confirmation to a wrong customer's number and every test would still pass.

The fix: in each demo that calls z-api__send_text, assert input.phone equals the expected number (the customer's phone passed in the prompt). For multi-recipient demos, assert the set of phones called matches the expected set.

Affected files:

• examples/whatsapp-installment-negotiation/skeleton.test.ts + live.test.ts
• examples/nfse-from-natural-language/skeleton.test.ts + live.test.ts

Gap 2 — NF-e / NFS-e payload fidelity

Only id and status are asserted on fiscal calls today. The actual payload — items[], CFOP, NCM, valor_total, valorServico, service codes — goes unchecked beyond what's implicit in the demo handler's canned response. A regression that drops items[], zeroes valor_total, or sends a CFOP/NCM unrelated to the prompt would ship green.

The fix: in each fiscal-issuing skeleton test, assert the call's input fields against the prompt. At minimum:

• items[] non-empty (or matches expected count).
• valor_total (or valorServico) equals the expected number.
• CFOP / NCM / service code matches the prompt's instruction.

Live tests can stay tolerant of LLM phrasing but should pin the numeric and enum fields.

Affected files:

• examples/nfse-from-natural-language/skeleton.test.ts + live.test.ts
• examples/whatsapp-installment-negotiation/skeleton.test.ts + live.test.ts
• examples/pix-nfse-skeleton/skeleton.test.ts + live.test.ts

Gap 3 — Output-message quality

Across every demo: expect(typeof r.message).toBe("string"). This passes for empty strings, hallucinated strings, English-only strings on a Portuguese flow, or strings that say "I cannot help with that." Buyer-facing copy is the whole point of a WhatsApp demo and is currently untested.

The fix: each skeleton test asserts r.message is non-empty and contains at least one of the expected semantic anchors for that flow (a customer name, an ID prefix, an amount, a Portuguese keyword like "confirma" / "pago" / "emitida"). Live tests use the same anchors with tolerant case-insensitive substring matching.

Affected files: every examples/*/skeleton.test.ts and examples/*/live.test.ts that exercises session.send().

Out of scope

• Adding a shared assertToolCall helper to factor out the per-test boilerplate — tracked separately as a scaffolding issue.
• Tightening live tests for nfse-from-natural-language and pix-nfse-skeleton to mirror their skeletons in general — tracked separately. This issue is the cross-demo overlay, not the per-demo work.
• New demos — these gaps apply to current demos only; new ones should land with these assertions already in place.

Acceptance

• All three gaps closed in every existing example that has a corresponding leg.
• Each new assertion would fail if the matching regression scenario happened (a wrong phone, a dropped items[], an empty r.message).
• Both npm run validate (aimock) and npm run validate:live (real Claude) still pass for every affected example.