step.request_parse: support application/x-www-form-urlencoded body parsing

[Copilot]

step.request_parse with parse_body: true only handled JSON bodies — form-encoded payloads (e.g. Twilio webhooks) were silently dropped and the body stream was consumed, breaking downstream steps like step.webhook_verify.

Changes

• Content-Type dispatch: After reading body bytes, strip any ; parameters from Content-Type and branch on application/x-www-form-urlencoded vs. the existing JSON path
• Form parsing: Use url.ParseQuery() to parse the body; single-value fields are exposed as string, multi-value fields as []string
• Raw body cache: Raw bytes are now stored in pc.Metadata["_raw_body"] whenever the body is freshly read from req.Body, regardless of Content-Type — ensuring downstream steps (e.g. HMAC-SHA1 verification) can always access the original payload
• Cache-first body read: If pc.Metadata["_raw_body"] is already populated by a prior step (e.g. step.webhook_verify), the cached bytes are used directly instead of reading the (possibly exhausted) req.Body

Example

Given a Twilio POST body Body=Hello&From=%2B15551234567&To=%2B15559876543:

- name: parse-request
  type: step.request_parse
  config:
    parse_body: true

Produces:

steps.parse-request.body.Body  = "Hello"
steps.parse-request.body.From  = "+15551234567"
steps.parse-request.body.To    = "+15559876543"
pc.Metadata["_raw_body"]       = []byte("Body=Hello&From=...")

Original prompt

---

This section details on the original issue you should resolve

<issue_title>step.request_parse: support application/x-www-form-urlencoded body parsing</issue_title>
<issue_description>## Current Behavior

step.request_parse with parse_body: true only supports JSON bodies. When the Content-Type is application/x-www-form-urlencoded, the step reads the body bytes via io.ReadAll() then calls json.Unmarshal(), which fails silently. As a result:

• output["body"] is never populated — the form field data is lost
• The body stream is consumed and unavailable for subsequent steps (e.g. step.webhook_verify)

This breaks webhook integrations that send form-encoded POST bodies, such as Twilio.

Requested Behavior

When parse_body: true is set and the Content-Type header is application/x-www-form-urlencoded, step.request_parse should:

1. Parse the body using url.ParseQuery() instead of json.Unmarshal()
2. Expose each form field in output["body"] as a string (first value) or []string (multiple values)
3. Cache the raw body bytes in pc.Metadata["_raw_body"] so subsequent steps (e.g. step.webhook_verify) can still access the original payload

Example

Given a Twilio webhook POST with body:

Body=Hello&From=%2B15551234567&To=%2B15559876543&MessageSid=SM1234

After step.request_parse with parse_body: true:

steps.<name>.body.Body       = "Hello"
steps.<name>.body.From       = "+15551234567"
steps.<name>.body.To         = "+15559876543"
steps.<name>.body.MessageSid = "SM1234"

And pc.Metadata["_raw_body"] should contain the original raw bytes so step.webhook_verify (which needs to re-read the body for HMAC-SHA1 validation) can still function.

Workaround

A custom step.form_parse step that reads _raw_body from pipeline metadata and parses via url.ParseQuery().</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes step.request_parse: support application/x-www-form-urlencoded body parsing #257

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Copilot]

Pull request overview

This PR extends step.request_parse to correctly parse application/x-www-form-urlencoded request bodies (common for webhook providers like Twilio) and adds tests for the new parsing behavior.

Changes:

• Add Content-Type dispatch to parse application/x-www-form-urlencoded bodies via url.ParseQuery (with single vs multi-value handling).
• Cache raw request body bytes into pc.Metadata["_raw_body"] (currently only on the form path).
• Add unit tests covering form parsing, multi-value fields, and Content-Type parameters (e.g. charset).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
module/pipeline_step_request_parse.go	Adds form-urlencoded parsing branch and attempts to cache raw body bytes.
module/pipeline_step_request_parse_test.go	Adds coverage for form-urlencoded parsing and raw body caching for that path.

[Copilot]

RequestParseStep always reads from req.Body when parse_body is enabled, but it never checks for an existing pc.Metadata["_raw_body"]. If another step (e.g. step.webhook_verify) has already read and cached the body, req.Body will be at EOF and this step will fail to populate output["body"]. Prefer using the cached _raw_body when present, and only fall back to reading req.Body when it is not cached.

[Copilot]

The raw body is cached into pc.Metadata["_raw_body"] only for application/x-www-form-urlencoded. For JSON (and any other Content-Type), this step still consumes req.Body and leaves _raw_body unset, which can break downstream steps that rely on the raw body cache (including step.webhook_verify / gitlab_parse_webhook patterns). Cache bodyBytes into _raw_body whenever you read it (before Content-Type dispatch), regardless of parsing success.

[Copilot]

There’s no test covering the common “body was already read and cached” flow: when pc.Metadata["_raw_body"] is set (e.g. by step.webhook_verify) and req.Body is empty/consumed, request_parse should still parse from the cached bytes. Adding a test for that scenario would prevent regressions and validate the intended raw-body-cache interoperability.

[intel352]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all three review items in commit 18f1d93:

1. Use cached _raw_body first: Before reading req.Body, the step now checks pc.Metadata["_raw_body"] and uses it if present — so a prior step consuming req.Body won't break parsing.
2. Always cache raw bytes: _raw_body is now set whenever body bytes are freshly read from req.Body, regardless of Content-Type (JSON or form-urlencoded).
3. New tests added: TestRequestParseStep_ParseBody_FormURLEncoded_CachedRawBody (parses from cached bytes when req.Body is empty) and TestRequestParseStep_ParseBody_JSON_CachesRawBody (verifies JSON path also populates _raw_body).