Support snapshotting original downstream and upstream header context for MCP Auth

[SavinduDimal]

Description

• Related Discussions: Expose Original Client Headers to Policies via Downstream/Upstream Context #2735
• This issue was created to track the progress of the fix for the bug reported in [Bug]: Set Header policy takes effect before mcp_auth policy #2548

[SavinduDimal]

Update

• During the initial discussions it was decided to go with the fix given in comment [1] with following new types in the SDK.

DownstreamContext

type DownstreamContext struct {
    Headers *Headers
}

UpstreamContext

type UpstreamContext struct {
	Headers *Headers
}

Changes to Context Structs

{
  "RequestHeaderContext": {
    "SharedContext": "(embedded)",           // existing
    "Headers":       "Headers",              // existing
    "Path":          "string",               // existing
    "Method":        "string",               // existing
    "Authority":     "string",               // existing
    "Scheme":        "string",               // existing
    "Vhost":         "string",               // existing
    "Downstream": {                          // NEW
      "Headers": "Headers"
    }
  },

  "RequestContext": {
    "SharedContext":  "(embedded)",          // existing
    "Headers":        "Headers",             // existing
    "Body":           "Body | null",         // existing
    "Path":           "string",              // existing
    "Method":         "string",              // existing
    "Authority":      "string",              // existing
    "Scheme":         "string",              // existing
    "Vhost":          "string",              // existing
    "Downstream": {                          // NEW
      "Headers": "Headers"
    }
  },

  "RequestStreamContext": {
    "SharedContext": "(embedded)",           // existing
    "Headers":       "Headers",              // existing
    "Path":          "string",               // existing
    "Method":        "string",               // existing
    "Authority":     "string",               // existing
    "Scheme":        "string",               // existing
    "Vhost":         "string",               // existing
    "Downstream": {                          // NEW
      "Headers": "Headers"
    }
  },

  "ResponseHeaderContext": {
    "SharedContext":   "(embedded)",         // existing
    "RequestHeaders":  "Headers",            // existing — headers forwarded to upstream
    "RequestBody":     "Body",               // existing
    "RequestPath":     "string",             // existing
    "RequestMethod":   "string",             // existing
    "ResponseHeaders": "Headers",            // existing
    "ResponseStatus":  "int",               // existing
    "Downstream": {                          // NEW
      "Headers": "Headers"
    },
    "Upstream": {                          // NEW
      "Headers": "Headers"
    }
  },

  "ResponseContext": {
    "SharedContext":   "(embedded)",         // existing
    "RequestHeaders":  "Headers",            // existing — headers forwarded to upstream
    "RequestBody":     "Body",               // existing
    "RequestPath":     "string",             // existing
    "RequestMethod":   "string",             // existing
    "ResponseHeaders": "Headers",            // existing
    "ResponseBody":    "Body | null",        // existing
    "ResponseStatus":  "int",               // existing
    "Downstream": {                          // NEW — original client request headers
      "Headers": "Headers"
    },
    "Upstream": {                          // NEW
      "Headers": "Headers"
    }
  },

  "ResponseStreamContext": {
    "SharedContext":   "(embedded)",         // existing
    "RequestHeaders":  "Headers",            // existing — headers forwarded to upstream
    "RequestBody":     "Body",               // existing
    "RequestPath":     "string",             // existing
    "RequestMethod":   "string",             // existing
    "ResponseHeaders": "Headers",            // existing
    "ResponseStatus":  "int",               // existing
    "Downstream": {                          // NEW — original client request headers
      "Headers": "Headers"
    },
    "Upstream": {                          // NEW
      "Headers": "Headers"
    }
  }
}

• Added above changes with PR: [SDK] Add Changes to Clone Downstream and Upstream Header Contexts #2631. And tested them with a custom policy to check the header context and previous live headers in both request and response phases (both header and body).
• Updated the existing policies which can be affected by the originally reported bug with new snapshot header contexts.
• However during the development process a new requirement came with the new field UpstreamInfo which needs to contain upstream URL and basepath. Hence the above introduced types and context changes will be replaced by the changes mentioned in the comment [1].

[1] - #2735 (comment)

[SavinduDimal]

Update

• Updated the PR: [SDK] Add Changes to Clone Downstream and Upstream Header Contexts #2631, with following request and response contexts.

RequestContext
 └── Downstream (DownstreamContext)
      └── Request (DownstreamRequest)
           └── Headers (Headers)
 └── Upstream (UpstreamRequestContext)
      ├── Name (string)
      ├── URL (string)
      └── BasePath (string)

ResponseContext
 └── Downstream (DownstreamContext)
      └── Request (DownstreamRequest)
           └── Headers (Headers)
 └── Upstream (UpstreamResponseContext)
      ├── Name (string)
      ├── URL (string)
      ├── BasePath (string)
      └── Response (UpstreamResponse)
           └── Headers (Headers)

• Currently we did not remove UpstreamInfo type but marked it as deprecated. Also added UpstreamRequestContext and UpstreamResponseContext to replace it in the future.
• Released the policies; JWT Auth, MCP Auth through PRs: Add support for request header snapshots and improve JWT Auth handling gateway-controllers#233, Add header snapshot functionality to mcp-auth policy gateway-controllers#235
• Tested the policies with the latest gateway.
• While testing with gateway v1.1.0 found the issue: [Bug]: API Platform Controller gateway build fails when a policy requires a newer Go version #2796
• Created issue: [Task]: Support snapshotting original downstream and upstream header context #2804 to track the policy SDK version upgrading task

[SavinduDimal]

Update

• Release jwt-auth v1.2.3 and mcp-auth v1.1.2 to release issue: [Bug]: API Platform Controller gateway build fails when a policy requires a newer Go version #2796.
• Tested the functionality of jwt-auth, mcp-auth and mcp-authz with gateway v1.1.0
  • Default flow is not affected
  • The reported bug is still reproducible
• Tested the policies with latest gateway v1.2.0 built images.