test(inbound): parser equivalence across Postmark / Mailgun / SES (NestJS)#22
Merged
test(inbound): parser equivalence across Postmark / Mailgun / SES (NestJS)#22
Conversation
Adds @nestjs-modules/mailer, nodemailer, handlebars runtime deps and @types/nodemailer for type support. Phase 0 / Task 0.1 of the public ticket system plan.
Fills the gap noted in the audit (no factories existed). Used by spec files across the public ticket feature work. Phase 0 / Task 0.3 of the public ticket system plan.
First-class identity for ticket requesters who may not have a host-app user account. Unique email index enables dedupe of repeat guests. Nullable userId allows linking to a host-app user later (see ContactService.promoteToUser in a follow-up commit). Phase 1 / Task 1.1 of the public ticket system plan.
Phase 1 / Task 1.2 of the public ticket system plan.
Provides: - findOrCreateByEmail(email, name?) — case-insensitive dedupe, trims whitespace, fills in blank existing name but never overwrites a non-blank one. - linkToUser(contactId, userId) — simple linkage. - promoteToUser(contactId, userId) — linkage + back-stamps ticket.requesterId on all tickets carrying the contactId. - findByEmail / findById — normalized lookups. Phase 1 / Tasks 1.4 + 1.5 of the public ticket system plan.
POST /escalated/widget/tickets now supports two paths:
1. Public — body.email (+optional name). Contact is resolved via
ContactService.findOrCreateByEmail and linked to the ticket.
requesterId is derived from the configured guestPolicy:
- unassigned → 0
- guest_user → options.guestPolicy.guestUserId
- prompt_signup → 0 + TicketSignupInviteEvent emitted
2. Legacy — body.requesterId (host-app authenticated user flow,
unchanged). No Contact resolution.
Missing both raises BadRequestException.
Adds ESCALATED_EVENTS.SIGNUP_INVITE + TicketSignupInviteEvent for the
signup-invite email listener (built in Phase 4).
Phase 2 / Tasks 2.2 + 2.4 of the public ticket system plan.
Adds PublicSubmitThrottleGuard with an in-memory sliding-window counter (10 submissions per hour per normalized email). Pass-through when body.email is absent, so the legacy requesterId path defers to the module-level ThrottlerGuard. Breach returns 429. Same-email case variants share a bucket. Timestamps older than the window are evicted lazily on next check. Known limitation: in-memory store scales to a single instance only; multi-instance deployments should swap for a Redis-backed store. Noted in source comment for a later phase. Phase 2 / Task 2.5 of the public ticket system plan.
…orkflowEngineService These were defined but never registered with TypeORM or exported from the module. Prerequisite for wiring WorkflowExecutor in a follow-up. Phase 3 / Task 3.0 of the public ticket system plan.
Implements the side-effecting half of the workflow system (distinct from WorkflowEngineService which only evaluates conditions). Actions shipped in this commit: - change_priority - add_tag (by slug, falls back to numeric id) - remove_tag (same resolution) - change_status (by slug, falls back to id; emits TICKET_STATUS_CHANGED) - set_department - assign_agent (writes activity log + emits TICKET_ASSIGNED) - add_note (internal reply) Deferred to follow-ups: send_webhook, add_follower, delay, assign_round_robin, send_notification, set_type. Not yet wired to events — WorkflowListener comes next. Phase 3 / Tasks 3.1-3.6 of the public ticket system plan.
…istener
Routing rules are now live. TicketCreatedEvent (and friends) fire a
chain:
listener → runner.runForEvent(triggerEvent, ticket)
→ workflowRepo.find({triggerEvent, isActive}) ASC position
→ for each: engine.evaluateConditions → log row → executor.execute
→ honor stopOnMatch, catch+log executor errors
Listener mappings:
TICKET_CREATED → ticket.created
TICKET_UPDATED → ticket.updated
TICKET_ASSIGNED → ticket.assigned
TICKET_STATUS_CHANGED → ticket.status_changed
TICKET_REPLY_CREATED → reply.created
WorkflowLog rows capture: matched condition, actions run, error message,
start/complete timestamps — sufficient for the existing Logs.vue UI.
Tests: 177 passing (+25 since Phase 2 — runner spec 6, listener spec 5,
executor spec 14).
Phase 3 / Task 3.11 of the public ticket system plan.
Proves the full chain wire-up: TicketService.create emits TICKET_CREATED → WorkflowListener fires → WorkflowRunnerService loads active workflows for ticket.created → conditions evaluated → executor.execute called with matching actions → WorkflowLog row written. Covers: match → executes, no workflows → no-op, conditions fail → log row without execution. Uses real EventEmitterModule (not mocked) with mocked repos and a mocked executor for observation. 3 tests, 180 total passing. Phase 3 / Task 3.12 of the public ticket system plan.
Templates rendered inline (HTML + plain text):
- renderTicketCreated: guest confirmation with reference number +
subject + body + optional portal link
- renderReplyPosted: agent reply notification (quotes the reply,
threads via In-Reply-To header)
- renderSignupInvite: prompt-signup flow CTA
EmailService methods:
- sendTicketCreated(to, ticket, contact, guestAccessToken)
- sendReplyPosted(to, ticket, reply, contact, guestAccessToken)
- sendSignupInvite(to, ticket, contact)
Every outbound message sets:
- Message-ID: <ticket-{id}[-reply-{replyId}]@{replyDomain}>
- X-Escalated-Ticket-Id: {id}
- Reply-To: reply+{id}.{hmac8}@{replyDomain} (for inbound routing)
Reply emails also set In-Reply-To + References → ticket's initial
Message-ID, so MUAs thread correctly.
No-ops silently when MailerService or options.mail is absent — the
module must boot without a mail transport configured.
Phase 4 / Task 4.3 of the public ticket system plan.
EmailListener subscribes to: - TICKET_CREATED → sendTicketCreated (when ticket.contactId set) - TICKET_REPLY_CREATED → sendReplyPosted (external replies only, not internal notes) - SIGNUP_INVITE → sendSignupInvite (only fires under prompt_signup policy) All handlers catch EmailService errors and warn-log — a mail outage never blocks ticket creation or reply posting (Phase 4 / Task 4.7). MailerModule.forRoot is registered conditionally — only when options.mail is present. Host apps that don't configure email still boot; EmailService no-ops silently. Tests: 205 passing (+25 in Phase 4 — message-id 11, email service 6, listener 8). Phase 4 / Tasks 4.1 + 4.4-4.7 of the public ticket system plan.
Captures every inbound webhook call with raw payload + parse summary + route outcome for diagnostics. Registered with TypeORM. Phase 5 / Task 5.1 of the public ticket system plan.
Provider-agnostic ParsedInboundEmail shape:
{ from, fromName, to, subject, textBody, htmlBody, messageId,
inReplyTo, references[] }
Postmark adapter pulls Message-ID / In-Reply-To / References from the
provider's Headers array (case-insensitive), handles missing fields,
falls back to FromFull.Email when flat From is absent.
Keeps the shape pluggable so Mailgun / SendGrid adapters can drop in
later (Phase 5b).
Phase 5 / Task 5.2 of the public ticket system plan.
Priority order for matching inbound email → existing ticket:
1. In-Reply-To header contains <ticket-{id}...@{domain}>
1b. References chain contains same
2. Envelope To is our signed reply+{id}.{hmac}@{replyDomain} address
3. Subject contains [TK-XXX] reference number
If none match, creates a new ticket via existing TicketService.create
path (reusing Contact dedupe by sender email). Missing sender → ignored.
All branches return a structured InboundRouteResult with matched/
created ids and error info for the audit log row.
Phase 5 / Task 5.3 of the public ticket system plan.
POST /escalated/webhook/email/inbound accepts Postmark inbound webhook
payloads and routes them via InboundRouterService (Phase 5 / 5.3):
- reply_added: existing ticket found via threading headers or
subject reference
- ticket_created: new ticket from sender's Contact
- ignored: malformed
- error: routing failure (audited)
Every request produces an InboundEmail audit row.
Signature guard (InboundWebhookSignatureGuard): constant-time compare
of X-Escalated-Inbound-Secret header against options.inbound.webhookSecret.
If secret is not configured, every request is rejected — prevents
accidental exposure.
Host app wires Postmark with Basic Auth → proxy adds the header.
Tests: 228 passing (+23 in Phase 5), 0 lint errors.
Phase 5 / Task 5.4 of the public ticket system plan.
…on fallback
Widget controller now calls SettingsService.getTyped('guest_policy', null)
first, falling back to options.guestPolicy. This means admins can change
guest policy at runtime via PUT /escalated/admin/settings (existing
endpoint, payload: { key: 'guest_policy', type: 'json', value: { ... } })
without a redeploy.
Module option remains the default for deployments that don't surface
the settings UI.
Phase 6 / Task 6.2 of the public ticket system plan.
…ation
New action type shared between Workflows and Macros (Phase 7). Takes a
template string with {{field}} placeholders, interpolates against the
ticket via WorkflowEngineService.interpolateVariables, inserts as an
external-visible reply.
Example: 'Hi! About {{subject}} (priority: {{priority}}) — on it.'
Phase 7 (Macros) reduced per the audit correction: Macro entity +
service + controllers already exist. This action type gives macros a
canned-reply capability that reuses WorkflowExecutor.
@nestjs-modules/mailer has lodash as a peer dependency but doesn't declare it in its peerDependencies — our local install resolved it transitively, but clean CI installs fail with: Cannot find module 'lodash' from 'node_modules/@nestjs-modules/mailer/dist/mailer.service.js' Adding lodash + @types/lodash explicitly. Fixes the CI test-(18/20/22) failures on PR #17.
CI's 'Check Prettier formatting' step was failing on 24 files added during Phases 0-9. Ran 'npm run format' (prettier --write) to bring them in line with the project's code style. No functional changes — pure whitespace / import-order / line-wrap normalization.
Brings the NestJS reference up to 2-provider parity with the greenfield ports (.NET / Spring / Go / Phoenix / Symfony). Before: the reference only had a Postmark parser and the controller hardcoded it. After: the controller picks the parser at request time via options.inbound.provider. MailgunInboundParser parses Mailgun's flat form-encoded body shape: sender / from / recipient / subject / body-plain / body-html / Message-Id / In-Reply-To / References. Extracts the display name from "Display Name <email@host>"-style from headers; strips surrounding quotes. InboundEmailController now injects both parsers and picks by provider label: - "mailgun" → MailgunInboundParser - "postmark" (default) → PostmarkInboundParser Module registers both parsers. 8 new Mailgun parser tests cover: happy path, threading headers + multi-id references, sender-fallback, bare-email no-display-name, quoted display name, To-vs-recipient fallback, empty body defaults, null payload. Updated controller spec picks up the new MailgunInboundParser dependency and adds a case asserting the provider switch routes through the right parser. Full suite: 240/240 tests pass.
Completes the 3-provider parity push on the NestJS reference (after
MailgunInboundParser in the previous commit). AWS SES receipt rules
publish to an SNS topic; host apps subscribe via HTTP and SNS POSTs
the envelope when options.inbound.provider is "ses".
Handles:
- Type=SubscriptionConfirmation — throws
SESSubscriptionConfirmationError (carrying topicArn +
subscribeUrl + token). Host apps catch it to surface the
URL they must GET out-of-band.
- Type=Notification — extracts threading metadata from
mail.commonHeaders, falls back to mail.headers array when a
field is absent.
- Best-effort MIME body extraction from the base64 content field:
handles single-part text/plain, text/html, multipart/alternative,
and quoted-printable + base64 transfer encoding. Hand-rolled
splitter — no external MIME lib dep.
InboundEmailController.pickParser now recognises `'ses'` and routes
to SESInboundParser.
13 SES parser tests (subscription confirmation sentinel, threading
metadata extraction, plain body decode, multipart body decode,
missing-content fallback, headers-array threading fallback, unknown
envelope type, missing/malformed Message). Full suite: 250/250 tests
pass (up from 240).
Brings the NestJS reference fully up to parity with the 5 greenfield
ports (.NET / Spring / Go / Phoenix / Symfony) — each now supports
all three major inbound providers.
Ports the per-framework downloaders (escalated-go#35 /
escalated-dotnet#29 / escalated-spring#32 / escalated-phoenix#41 /
escalated-symfony#37) to the NestJS reference. Brings the reference
to full parity with what the ports ship.
AttachmentDownloader:
- download(pending, ticketId, replyId?) — fetch + persist via an
AttachmentStorage adapter + new Attachment row on the TypeORM
repository.
- downloadAll continues past per-attachment failures; returns a
parallel AttachmentDownloadResult[] (persisted|error per input).
- AttachmentDownloaderOptions: maxBytes size cap (throws
AttachmentTooLargeError) + optional HTTP basic auth (Mailgun
API-key URLs) + overridable `fetch` for tests.
- safeFilename strips path separators so "../../etc/passwd" →
"passwd" — crafted names can't escape the storage root.
- Response Content-Type fallback when the pending record's
contentType is blank.
LocalFileAttachmentStorage:
- Reference AttachmentStorage that writes to local disk with
timestamp + random-suffix filename prefixing to avoid
collisions under concurrent writes.
- Host apps with S3 / GCS / Azure implement AttachmentStorage
themselves and bind via the ATTACHMENT_STORAGE DI token.
17 tests cover: happy path, reply-attached variant, HTTP errors,
oversize rejection, basic-auth header, missing-URL guard,
Content-Type fallback, safeFilename table, downloadAll
partial-failure batching, LocalFileStorage write / empty-root
rejection / unique path per call.
Full suite: 267/267 tests pass (up from 250).
…stJS) Closes the parser-equivalence test rollout across all 6 frameworks: NestJS reference, plus the 5 greenfield ports (#37 go, #31 dotnet, #34 spring, #43 phoenix, #39 symfony). Two Jest cases: - normalizes to the same threading metadata: from / to / subject / inReplyTo / references match across all three parsers for a single logical reply email. Failure messages are tagged with the offending parser so a regression points at the right file. - produces the same body text: Postmark + Mailgun forward directly; SES decodes from base64 MIME (test uses `contains` since SES may include a trailing newline). Shared LogicalEmail interface + *Payload builders mean adding a fourth provider validates against the existing three for free. Full suite: 269/269 tests pass (up from 267).
mpge
force-pushed
the
test/parser-equivalence
branch
from
0a9b349 to
538b70f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the parser-equivalence test rollout across all 6 frameworks — NestJS reference plus the 5 greenfield ports:
Tests
Two Jest cases:
Shared `LogicalEmail` interface + `*Payload` builders mean adding a fourth provider validates against the existing three for free.
Full suite: 269/269 tests pass (up from 267).
Stacked PR
Based on `feat/attachment-downloader` (#21). Merge order: #17 → #18 → #19 → #20 → #21 → this PR.