Skip to content

Tighten live tests for nfse-from-natural-language and pix-nfse-skeleton to mirror the skeleton assertions #42

Description

@dangazineu

Summary

The live.test.ts files in examples/nfse-from-natural-language/ and examples/pix-nfse-skeleton/ are much coarser than their tightened skeleton.test.ts counterparts. They would pass through real regressions that the skeleton tests catch — defeating the purpose of having a live-LLM smoke gate at all.

The bar to aim for: roughly the assertion density that #34 set for the WhatsApp installment negotiation skeleton. Live tests stay tolerant of LLM probabilism (call ordering, extra retries, message phrasing) but should pin the wire-contract fields that don't depend on the model's phrasing.

Concrete gaps

examples/nfse-from-natural-language/live.test.ts

Today it only checks nfseCalls.length >= 1, every dispatched call has status === "success", and id matches /^nfse_/. It would pass even if:

  • Claude issued one NFS-e with the wrong service code.
  • Claude got the valor wildly wrong.
  • The WhatsApp delivery step was skipped entirely.

The skeleton test pins all of these. The live test should at least pin:

  • Two NFS-e callsnfseCalls.length === 2 (or >= 2 if you want to tolerate retries).
  • Per-call valor fidelity — one of [2800, 1200] per call, both present across the two calls.
  • LC 116 service codes — one of ["1.05", "17.01"] per call, both present across the two calls.
  • Demo-fixture id shapeid matches /^nfse_demo_/, not just /^nfse_/.
  • WhatsApp delivery presence — at least one z-api__send_text call whose body references both NFS-e IDs (or both PDF URLs). Tolerate phrasing; pin the IDs.

examples/pix-nfse-skeleton/live.test.ts

Today it checks asaasCalls.length >= 1, nfeCalls.length >= 1, and every call succeeded. It would pass even if:

  • Asaas billing type was wrong (CREDIT_CARD instead of PIX).
  • The charge amount was wrong (anything, since the test never reads value).
  • The QR code envelope was malformed.
  • The NFS-e was issued for a wildly different valorServico.

The tightened skeleton pins pay_demo_* regex, billingType: "PIX", value: 150.0, the BR-Code envelope /^00020126/, per-step success flag, step-4 numero + valorServico typeof. The live test should mirror at minimum:

  • BR-Code envelope — at least one Asaas QR-fetching call returns a payload whose BR-Code starts with /^00020126/.
  • Per-call success — each dispatched call individually carries status === "success" (the test already does this; keep it).
  • Billing type — at least one asaas__create_payment (or create_charge) call carries billingType: "PIX".

Out of scope

  • Asserting the exact NF-e payload (items[], CFOP, NCM, valor_total) — that's a cross-demo gap tracked separately and applies to all WhatsApp/fiscal demos, not just these two.
  • Phone-number routing assertions — also tracked separately as a cross-demo gap.
  • Asserting z-api__send_text presence on the WhatsApp installment negotiation live test — tracked as a separate follow-up that feat(examples): WhatsApp installment negotiation (multi-turn session.send + Asaas preview) #34 explicitly flagged.
  • Changing the skeleton tests — they're the bar; live tests catch up to them.

Acceptance

  • examples/nfse-from-natural-language/live.test.ts and examples/pix-nfse-skeleton/live.test.ts updated.
  • Both live tests still pass against real Claude (probabilism-tolerant — no exact phrasing checks).
  • The new assertions would fail if the corresponding skeleton-test scenario regressed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions