step.http_call: add oauth2 config key and Salesforce instance_url support

[Copilot]

• Investigate existing step.http_call OAuth2 implementation
• Add oauth2 config key as alternative to auth (Salesforce-friendly syntax)
• Add instance_url caching in oauthCacheEntry
• Parse and cache instance_url from token response in doFetchToken
• Inject instance_url into template data for URL resolution in Execute
• Include instance_url in step output
• Update step.http_call schema to document oauth2 field
• Add tests for new oauth2 config syntax
• Add tests for Salesforce instance_url support
• Extract buildOAuthConfig shared helper to eliminate duplicate parsing logic between auth and oauth2 blocks
• Fix 401-retry path to refresh pc.Current["instance_url"] and re-resolve URL template after token refresh (instance_url may change)
• Fix TestHTTPCallStep_OAuth2_InstanceURL to actually test token response parsing (not pre-populated cache)
• Add TestHTTPCallStep_OAuth2_Retry401_RefreshesInstanceURL test for 401 retry with instance_url re-resolution
• Fix schema URL field description to clarify instance_url is available for all OAuth2 client_credentials (not just oauth2 block)
• Document auth-takes-precedence-over-oauth2 in schema description
• Run linter + tests - all pass

Original prompt

---

This section details on the original issue you should resolve

<issue_title>step.http_call: OAuth2 client_credentials token caching for outbound API calls</issue_title>
<issue_description>## Use Case

When using step.http_call to call external APIs that require OAuth2 authentication (e.g., Salesforce REST API), the pipeline needs to:

1. Obtain an access token via client_credentials grant before making the API call
2. Cache the token for its lifetime to avoid per-request token fetches
3. Automatically refresh when the token expires
4. Retry the API call with a fresh token on 401 responses

Current Workaround

Currently, the only way to call OAuth2-protected external APIs is to chain two step.http_call steps:

1. First call to the token endpoint to get an access token
2. Second call using the token in the Authorization header

This has several problems:

• Token is fetched on every pipeline execution (no caching)
• No automatic retry on 401
• Adds latency to every request
• Token endpoint gets unnecessary load

Proposed Solution

Add OAuth2 client_credentials support to step.http_call configuration:

- name: call_salesforce
  type: step.http_call
  config:
    url: "https://org.salesforce.com/services/data/v62.0/sobjects/Application__c"
    method: POST
    oauth2:
      grant_type: client_credentials
      token_url: "${SALESFORCE_TOKEN_URL}"
      client_id: "${SALESFORCE_CLIENT_ID}"
      client_secret: "${SALESFORCE_CLIENT_SECRET}"
      scopes: ["api"]
    headers:
      Content-Type: "application/json"
    body:
      Lead__r:
        Okta_user_Id__c: "{{.steps.parse.body.okta_id}}"

The engine would:

• Fetch and cache the access token internally
• Set Authorization: Bearer <token> header automatically
• Refresh on expiry or 401 response
• Support instance_url from the token response (Salesforce pattern)

Context

This is needed for Chimera's Salesforce integration where form submissions trigger case creation via the Salesforce REST API. The Salesforce token endpoint returns both access_token and instance_url, and the instance URL is needed to construct the API endpoint.

🤖 Generated with Claude Code</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes step.http_call: OAuth2 client_credentials token caching for outbound API calls #241

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[Copilot]

Pull request overview

This PR enhances step.http_call to better support OAuth2 client-credentials flows (notably Salesforce) by adding an ergonomic top-level oauth2 config and propagating instance_url from the token response into template resolution and step outputs.

Changes:

• Added a top-level oauth2 config block (as an alternative to auth.type: oauth2_client_credentials) with basic validation.
• Cached instance_url alongside the access token and injected it into pc.Current before URL template resolution; also included it in step output.
• Added unit tests covering the new config path, default/invalid grant_type handling, and instance_url behavior; updated schema docs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
module/pipeline_step_http_call.go	Adds instance_url caching/accessors, parses instance_url from token responses, supports top-level oauth2 config, and reorders token fetch before URL template resolution.
module/pipeline_step_http_call_test.go	Adds tests for the new oauth2 config key, grant_type validation/defaulting, and instance_url injection/caching behavior.
schema/module_schema.go	Documents the new oauth2 config block and notes instance_url usage in URL templates.

[Copilot]

The 401-retry path now returns instance_url from the refreshed token, but the retry request is still sent to the pre-refresh resolvedURL and uses whatever pc.Current contained before refresh. If instance_url changes between token responses (Salesforce can), the retry may hit the wrong host. After doFetchToken on 401, refresh pc.Current["instance_url"] from the cache and re-resolve the URL/template-dependent fields before building the retry request.

[Copilot]

The URL field description suggests {{ .instance_url }} is only available when the top-level oauth2 block is used, but the code populates instance_url for any OAuth2 client_credentials auth (including auth.type=oauth2_client_credentials). Consider rewording to avoid implying the feature is limited to the oauth2 config key.

Suggested change

	{Key: "url", Label: "URL", Type: FieldTypeString, Required: true, Description: "Request URL (supports {{ .field }} templates; use {{ .instance_url }} when oauth2 is configured with an instance_url-returning endpoint)", Placeholder: "https://api.example.com/{{ .resource }}"},
	{Key: "url", Label: "URL", Type: FieldTypeString, Required: true, Description: "Request URL (supports {{ .field }} templates; {{ .instance_url }} is available when OAuth2 client_credentials auth uses a token endpoint that returns instance_url)", Placeholder: "https://api.example.com/{{ .resource }}"},

[Copilot]

The new top-level oauth2 parsing largely duplicates the existing auth.type=oauth2_client_credentials parsing (scopes parsing, cacheKey derivation, required fields). This duplication increases the risk the two config paths drift (e.g., future additions like extra params). Consider extracting a shared helper to build oauthConfig/oauthEntry from a generic map, or normalize oauth2 into an authCfg and reuse the existing branch.

[Copilot]

In TestHTTPCallStep_OAuth2_InstanceURL, the cache is pre-populated (hs.oauthEntry.set(...)), so getToken() never calls the token server and the test doesn't actually validate parsing instance_url from the token response. To better match the test name/comment, consider letting the step fetch the token (have the tokenSrv return apiSrv.URL as instance_url) and assert the token endpoint was hit, instead of seeding the cache manually.

[Copilot]

Schema docs for the new oauth2 field don't mention how it interacts with the existing auth block (the factory only applies oauth2 when auth is absent). Consider documenting the precedence/mutual-exclusion so users don't accidentally specify both and wonder why oauth2 is ignored.

Suggested change

	{Key: "method", Label: "Method", Type: FieldTypeSelect, Options: []string{"GET", "POST", "PUT", "PATCH", "DELETE"}, DefaultValue: "GET", Description: "HTTP method"},
	{Key: "headers", Label: "Headers", Type: FieldTypeMap, MapValueType: "string", Description: "Request headers (values support templates)"},
	{Key: "body", Label: "Body", Type: FieldTypeJSON, Description: "Request body (supports templates). For POST/PUT without body, sends pipeline context."},
	{Key: "timeout", Label: "Timeout", Type: FieldTypeString, DefaultValue: "30s", Description: "Request timeout duration", Placeholder: "30s"},
	{Key: "oauth2", Label: "OAuth2", Type: FieldTypeJSON, Description: "OAuth2 client_credentials configuration. Tokens are cached and refreshed automatically. Fields: grant_type (default: client_credentials), token_url, client_id, client_secret, scopes. When the token endpoint returns instance_url (Salesforce pattern), it is injected as {{ .instance_url }} for URL templates and included in the step output."},
	{Key: "method", Label: "Method", Type: FieldTypeSelect, Options: []string{"GET", "POST", "PUT, "PATCH", "DELETE"}, DefaultValue: "GET", Description: "HTTP method"},
	{Key: "headers", Label: "Headers", Type: FieldTypeMap, MapValueType: "string", Description: "Request headers (values support templates)"},
	{Key: "body", Label: "Body", Type: FieldTypeJSON, Description: "Request body (supports templates). For POST/PUT without body, sends pipeline context."},
	{Key: "timeout", Label: "Timeout", Type: FieldTypeString, DefaultValue: "30s", Description: "Request timeout duration", Placeholder: "30s"},
	{Key: "oauth2", Label: "OAuth2", Type: FieldTypeJSON, Description: "OAuth2 client_credentials configuration. Tokens are cached and refreshed automatically. Fields: grant_type (default: client_credentials), token_url, client_id, client_secret, scopes. When the token endpoint returns instance_url (Salesforce pattern), it is injected as {{ .instance_url }} for URL templates and included in the step output. If both the legacy auth block and oauth2 are configured, auth takes precedence and oauth2 is ignored.",},

[intel352]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all changes from the review thread in commit a988dec:

1. Extracted buildOAuthConfig shared helper — eliminates the duplicate scopes parsing, required-field validation, and cache-key derivation between the auth and oauth2 config paths. Both now delegate to this single function.

2. Fixed 401-retry path — after doFetchToken on 401, the code now re-injects instance_url from the refreshed token into pc.Current and re-resolves the URL template before building the retry request. Added TestHTTPCallStep_OAuth2_Retry401_RefreshesInstanceURL to cover this scenario.

3. Fixed TestHTTPCallStep_OAuth2_InstanceURL — removed the pre-populated cache seeding. The token server now returns apiSrv.URL as instance_url, the step fetches the token live, and the test asserts the token endpoint was called exactly once.

4. Updated schema URL field description — now says "available when OAuth2 client_credentials auth uses a token endpoint that returns instance_url" (not tied to the oauth2 key specifically).

5. Updated oauth2 schema field description — documents that if both auth and oauth2 are specified, auth takes precedence and oauth2 is ignored.