Skip to content

fix(frontend): scope archived pod-log reads to the authorized namespace#104

Open
herikwebb wants to merge 18 commits into
masterfrom
security/fix-podlogs-archive-namespace-scope-20260724132914-12760
Open

fix(frontend): scope archived pod-log reads to the authorized namespace#104
herikwebb wants to merge 18 commits into
masterfrom
security/fix-podlogs-archive-namespace-scope-20260724132914-12760

Conversation

@herikwebb

Copy link
Copy Markdown
Owner

Summary

The ml-pipeline-ui pod-logs handler authorizes access against the caller-supplied podnamespace, but the archive fallback (getPodLogsStreamFromArchive) derives the object-store key entirely from the caller-supplied podname/createdat via the configured keyFormat. The shipped default keyFormat is not namespace-scoped, so in multi-user mode the namespace access check does not constrain which archived log is returned, and an authenticated tenant could read another tenant's archived pod logs from the shared bucket.

Change

In createPodLogsMinioRequestConfig, fail closed when auth is enabled:

  • Require a non-empty authorized namespace and require the keyFormat to embed {{workflow.namespace}}, so the resolved key stays confined to the authorized namespace prefix. If the key template is not namespace-scoped, refuse the archive read instead of serving an unscoped key.
  • Reject any .. path segment in the resolved key so a caller-controlled podname cannot climb out of the namespace prefix (the existing regex allows . and /).

Behavior is unchanged when auth is disabled (single-user / standalone).

Tests

Adds frontend/server unit tests for the namespace-scoped key, both fail-closed paths, and the .. rejection. workflow-helper vitest suite passes; tsc --noEmit and Prettier are clean on the changed files.

Tracking / disclosure

Tracking issue: #103

Per kubeflow/pipelines SECURITY.md, coordinated disclosure should go through the upstream private channel (a GitHub Security Advisory on kubeflow/pipelines or ksc@kubeflow.org); this fork PR is the review/tracking record and is not for merge here.


Generated by Claude Code

Herik Webb and others added 15 commits July 24, 2026 13:28
Signed-off-by: Herik Webb <herik@shardplate.local>
Signed-off-by: Herik Webb <herik@shardplate.local>
Add a pull request watch policy so that any agent (including scheduled
routines) subscribes to a PR immediately after opening it, handing it to
the persistent watcher for review-feedback and CI follow-up until the PR
is merged or closed.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
…tter

Codify that every commit must be authored, committed, and signed off by the
human submitter rather than an agent identity. Automated environments default
the git identity to the agent (e.g. Claude <noreply@anthropic.com>), so a bare
'git commit -s' signs off as the agent and breaks the DCO requirement. Document
setting the repo-local identity to the submitter, verifying author and sign-off
before pushing, and reusing the submitter identity for follow-up commits on a
PR they already authored.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
gpt-5.5 is a reasoning model whose reasoning tokens count against
max_output_tokens, so the previous 3000 budget was intermittently exhausted
before any review text was produced, failing the review job with 'OpenAI API
response did not include review text'. Raise the default to 16000 and make it
overridable via the OPENAI_MAX_OUTPUT_TOKENS repository variable.

Also export the variable in review-pr.sh so the python child process actually
inherits it — previously it was set without export, so the child always used
its own hardcoded default regardless of the shell value.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Raising the token budget means a review can exceed GitHub's 65536-character
comment body limit, which would make 'gh pr comment' fail after a successful
model call. Truncate the body (with margin) and append a notice when it is too
large instead of failing to post the review.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
`resolveFilePathOnVolume` in `frontend/server/utils.ts` composed the final
filesystem path with `path.join(volumeMountPath, filePathInVolume)`, which
collapses `..` segments. `getArtifactsHandler` in
`frontend/server/handlers/artifacts.ts` validates the `bucket` query param
with `isAllowedResourceName` but only length-caps the `key` param and hands
it to `getVolumeArtifactsHandler` → `findFileOnPodVolume` →
`resolveFilePathOnVolume` unchanged.

Combined with a well-known volume name mounted on the ml-pipeline-ui pod
(for example `config-volume` from
`manifests/kustomize/base/pipeline/ml-pipeline-ui-deployment.yaml`), an
authenticated Kubeflow user could pass
`?source=volume&bucket=config-volume&key=../../var/run/secrets/kubernetes.io/serviceaccount/token`
through the artifacts auth middleware — which only checks that they can
access their own namespace — and stream the ml-pipeline-ui pod's service
account token (or any other file the pod's filesystem contains).

Normalize the joined path with `path.resolve` and require that it stays
under the resolved `volumeMountPath` (equal to it or under
`volumeMountPath + '/'`). This blocks both `../` escapes when no `subPath`
is set and `subPath + '/../../'` escapes when `subPath` is configured. Add
two `resolveFilePathOnVolume` regression tests covering both variants.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
…eads

The lexical path.resolve containment check in resolveFilePathOnVolume stops
'..' traversal but not a symlink inside the mounted volume pointing outside it
(e.g. 'link -> /etc/passwd'), which fs.createReadStream would follow at read
time.

Add resolveRealPathWithinMount, which resolves the real (symlink-free) path
immediately before the read and re-verifies it stays within the volume mount,
minimizing the TOCTOU window by returning the fully resolved path to open.
findFileOnPodVolume now also returns the resolved volume mount path so the
read site can perform this check; the extra tuple element is backward
compatible with existing two-element destructures. getVolumeArtifactsHandler
calls the resolver before stat/createReadStream and reads the returned safe
path. When the path cannot be resolved (missing file) the original path is
returned so existing not-found behavior is preserved.

Extract a shared isPathInsideMount helper used by both the lexical and the
realpath containment checks. Add resolveRealPathWithinMount tests covering an
in-mount file, a symlink escaping the mount, and an unresolvable path.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
The previous fix resolved the real path and re-checked containment, but the
handler then reopened that path with stat()/createReadStream(). An attacker
able to write the volume could swap the file for a symlink between the check
and the open, and both calls follow symlinks, so the read could still escape.

Replace resolveRealPathWithinMount with openRealFileWithinMount, which opens
the file once, verifies the opened descriptor's real target (via
/proc/self/fd on Linux, falling back to path resolution elsewhere) is inside
the mount, and returns the handle. getVolumeArtifactsHandler streams from that
same handle instead of reopening by name, so validation and read act on one fd
and cannot be raced. Open errors (missing file) still propagate to preserve the
existing not-found behavior.

Update tests to cover the open-and-validate handle, symlink rejection, and
missing-file propagation.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Match the notebook-intelligence PR reviewer:
- Trigger on pull_request instead of pull_request_target and check out the PR
  head directly (fork PRs get a read-only token and no secrets, so running the
  diff-only reviewer on the checked-out head is safe and simpler).
- Emit a machine-readable VERDICT (APPROVE / CHANGES_REQUESTED) and fail the
  check on a non-APPROVE verdict, so a green 'review' check means the reviewer
  signed off. This is what an automated promotion can key off.
- Add an 'override-review-gate' label escape hatch that relaxes only the exit
  status; the review still runs and comments.
- Use a three-dot (merge-base) diff range with a two-dot fallback on shallow
  clones, so the reviewer sees what the PR introduces on merge.

Keeps the fork's OPENAI_MAX_OUTPUT_TOKENS=16000 override: gpt-5.5 is a reasoning
model and 3000 was easily exhausted, producing empty responses that would now
fail the review gate.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Add a workflow_dispatch action + scripts/promote-upstream.sh that opens a
cross-fork PR from a chosen fork branch into kubeflow/pipelines, using a
human PAT (secrets.UPSTREAM_PAT) since the default GITHUB_TOKEN cannot write
upstream. It resolves the upstream default branch, refuses empty promotions,
guards against leaking fork-only automation files into the upstream diff, and
by default requires the branch's fork PR review gate to be green before
promoting (override with require_review_pass=false). Change detection uses
GitHub's server-side compare API so it is independent of local clone depth.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
promote-upstream.sh used GitHub's compare API only and never rebased, so a
fix branch cut from the fork's default branch dragged the fork-only
automation (pr-review.yml, this script, AGENTS.md, ...) into the upstream
compare and tripped the fork-only guard — the same branch could not serve
both the fork review PR and the upstream promotion.

Add an optional FORK_BASE input: when set (e.g. master), replay only the
commits the branch adds over that base onto the current upstream tip,
push the result as a script-managed derived branch, and promote that. The
review gate still keys off the original fork PR branch. Blank FORK_BASE
preserves the previous promote-as-is behavior.
The derive step now cherry-picks with --signoff so the promoted commits
carry a Signed-off-by trailer for the submitter identity, satisfying
kubeflow/pipelines' DCO requirement without a manual re-sign.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
With more than one fix commit GitHub falls back to the branch name for the
PR title and an empty description. Default the promotion PR title/body from
the primary (first) fix commit instead of the branch tip, list the included
commits, and append the sign-off / title-convention checklist. Add a body
input to the workflow so it can be overridden. No squashing: upstream
squash-merges anyway, so the PR title/body is what becomes the merge commit.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
The ml-pipeline-ui pod-logs handler authorizes access against the
caller-supplied `podnamespace`, but the archive fallback
(getPodLogsStreamFromArchive) derives the object key entirely from the
caller-supplied `podname`/`createdat` via the configured keyFormat. The
default keyFormat is not namespace-scoped, so in multi-user mode an
authenticated tenant could read another tenant's archived pod logs from
the shared bucket by naming the target pod — the namespace access check
does not constrain the archive object key.

Fail closed in multi-user mode: require the keyFormat to embed
{{workflow.namespace}} (and a non-empty namespace) so the resolved key
stays confined to the authorized namespace prefix, and reject any '..'
path segment so a caller-controlled podname cannot climb out of that
prefix. Behavior is unchanged when auth is disabled.

Adds unit tests for namespace-scoped keys, the fail-closed paths, and
the '..' rejection.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
herikwebb added a commit that referenced this pull request Jul 24, 2026
Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: CHANGES_REQUESTED

Changed files:

frontend/server/handlers/pod-logs.ts
frontend/server/workflow-helper.test.ts
frontend/server/workflow-helper.ts

Review result:

Automated review findings

High — frontend/server/workflow-helper.ts

Namespace scoping check only requires {{workflow.namespace}} to appear, not that it forms an unambiguous namespace prefix/path segment.

The new multi-user guard rejects key formats that omit {{workflow.namespace}}, but a key format can still pass while not being safely scoped to the authorized namespace. For example, formats that concatenate namespace with another attacker-controlled field or use a non-boundary delimiter can collide with another namespace’s archived log key:

artifacts/{{workflow.namespace}}-{{pod.name}}
artifacts/{{workflow.namespace}}{{pod.name}}

Because podName is supplied by the request and interpolated into the key, a tenant in namespace foo may be able to construct keys that overlap with logs under namespace-like prefixes such as foo-bar, depending on the configured keyFormat.

The comment says the key must stay confined to the authorized prefix, but key.includes('{{workflow.namespace}}') does not enforce a prefix boundary. Consider requiring the namespace placeholder to be a complete path segment, e.g. matching (^|/){{workflow.namespace}}(/|$), and rejecting key formats where it is adjacent to other dynamic/static characters. Add tests for adjacent/concatenated namespace templates.

Address review feedback: the multi-user archive-key guard previously only
checked that {{workflow.namespace}} appeared anywhere in the keyFormat.
A template placing it adjacent to a caller-controlled field (e.g.
"{{workflow.namespace}}{{pod.name}}" or "{{workflow.namespace}}-{{pod.name}}")
would pass while still letting a tenant in namespace "foo" construct keys
overlapping "foobar"/"foo-bar".

Require {{workflow.namespace}} to be a complete '/'-delimited path segment
(matching (^|/){{workflow.namespace}}(/|$)) so the authorized namespace forms
an unambiguous prefix/segment. Add tests for concatenated and delimiter-adjacent
templates (rejected) and a trailing namespace segment (accepted).

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>

Copy link
Copy Markdown
Owner Author

Thanks — good catch. Fixed in f3c65fe.

includes('{{workflow.namespace}}') was too weak; a keyFormat placing the namespace tag adjacent to the caller-controlled {{pod.name}} (e.g. {{workflow.namespace}}{{pod.name}} or {{workflow.namespace}}-{{pod.name}}) would pass while letting a tenant in foo reach keys under foobar/foo-bar. The guard now requires {{workflow.namespace}} to be a complete /-delimited path segment, matching (^|/){{workflow.namespace}}(/|$).

Added tests for the concatenated and delimiter-adjacent templates (both rejected) and for a trailing namespace segment (accepted). workflow-helper vitest (18 tests), tsc --noEmit, and Prettier are all clean.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: CHANGES_REQUESTED

Changed files:

frontend/server/handlers/pod-logs.ts
frontend/server/workflow-helper.test.ts
frontend/server/workflow-helper.ts

Review result:

Automated review findings

  • Severity: High
    Files: frontend/server/handlers/pod-logs.ts, frontend/server/workflow-helper.ts
    Issue: Namespace scoping is enforced only for the final configured archive fallback, not for the workflow-status archive path.
    Rationale: In multi-user mode the handler now passes authEnabled into createPodLogsMinioRequestConfig, which rejects key formats that are not scoped by {{workflow.namespace}}. However, composePodLogsStreamHandler still tries getPodLogsStreamFromWorkflow before that fallback. getPodLogsStreamFromWorkflow trusts the main-logs artifact s3.key from workflow status and reads it using the server/shared object-store credentials, without validating that the key is confined to the authorized namespace. If a tenant can influence the workflow artifact key or artifact repository configuration for their namespace, they can potentially point the workflow-status log artifact at another namespace’s archived log object and bypass the new fail-closed check entirely.
    Suggested fix: Apply equivalent namespace-scoping validation to workflow-derived logKey when authEnabled is true, or pass authEnabled through and refuse workflow-status log retrieval unless the resolved key is confined to the authorized namespace. Add a regression test covering the handler/workflow-status fallback path with an unscoped or cross-namespace artifact key.

Address review feedback: the previous change hardened only the configured
archive fallback, but composePodLogsStreamHandler attempts
getPodLogsStreamFromWorkflow first. That path reads the main-logs artifact
s3.key from the workflow status and fetches it with the shared object-store
credentials, without confining the key to the authorized namespace. A tenant
who can influence their namespace's artifact-repositories keyFormat could point
the recorded main-logs key at another namespace's log object and read it,
bypassing the fallback guard.

Thread authEnabled into getPodLogsMinioRequestConfigfromWorkflow and, in
multi-user mode, require the workflow-recorded log key to contain the authorized
namespace as a complete '/'-delimited path segment (rejecting '..'), mirroring
the archive-fallback check. Bind authEnabled where the handler composes the
workflow-status stream. Adds regression tests for an unscoped/cross-namespace
key (rejected) and a namespace-scoped key (accepted).

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>

Copy link
Copy Markdown
Owner Author

Good catch — fixed in 7b2133d.

You're right that composePodLogsStreamHandler tries getPodLogsStreamFromWorkflow before the configured archive fallback, and that path read the main-logs s3.key from the workflow status with the shared object-store credentials without confining it to the authorized namespace. A tenant able to shape their namespace's artifact-repositories keyFormat could point the recorded key at another namespace's object and bypass the fallback guard.

authEnabled is now threaded into getPodLogsMinioRequestConfigfromWorkflow, which in multi-user mode requires the workflow-recorded log key to contain the authorized namespace as a complete /-delimited path segment (and rejects ..), mirroring the archive-fallback check. The handler binds authEnabled where it composes the workflow-status stream. Added regression tests for an unscoped/cross-namespace key (rejected) and a namespace-scoped key (accepted). 20 workflow-helper tests pass; tsc --noEmit and Prettier are clean.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: CHANGES_REQUESTED

Changed files:

frontend/server/handlers/pod-logs.ts
frontend/server/workflow-helper.test.ts
frontend/server/workflow-helper.ts

Review result:

Automated findings

High — frontend/server/workflow-helper.ts

The new multi-user confinement check for workflow-recorded log keys only verifies that the authorized namespace appears anywhere as a /-delimited segment:

const segments = logKey.split('/');
if (!namespace || !segments.includes(namespace) || segments.includes('..')) { ... }

This does not prove the object key is actually scoped to that namespace. A malicious or compromised workflow/artifact repository configuration could point main-logs at another namespace’s object whose key happens to contain the authorized namespace as another segment, such as a workflow name or pod-derived segment. The same weak “namespace can be any segment” assumption is also codified for configured archive key formats, including a test that accepts the namespace as a trailing segment.

Because this path reads from the object store using shared/server credentials, the check should validate a deterministic namespace-scoped prefix or otherwise recompute/verify the key from a trusted template where {{workflow.namespace}} appears before any user-controlled fields. Merely requiring segments.includes(namespace) is not sufficient isolation for multi-user mode.

Address review feedback: requiring {{workflow.namespace}} to appear merely as
some '/'-delimited segment did not prove the key was scoped to that namespace —
a key from another namespace could coincidentally contain the authorized
namespace as a later segment (e.g. a workflow-name or pod-derived segment).

Introduce a shared namespaceScopedKeyPrefix() helper that returns the deterministic
namespace-scoped prefix only when {{workflow.namespace}} is a complete '/'-delimited
segment ahead of every caller-influenced field (pod name, workflow name, creation
timestamp), and null otherwise. Both archived-log paths now use it:

- createPodLogsMinioRequestConfig fails closed unless the (operator/config) keyFormat
  yields such a prefix.
- getPodLogsMinioRequestConfigfromWorkflow receives the trusted operator keyFormat and
  requires the workflow-recorded key to begin with the derived namespace prefix (and
  rejects '..'), so a tenant-influenced artifact key cannot point at another namespace's
  object.

Update tests: a template placing a caller field before the namespace tag (including a
trailing namespace segment) is now rejected; add coverage for a bounded prefix segment,
the workflow-status prefix match, and fail-closed when the trusted keyFormat is not
namespace-scoped.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>

Copy link
Copy Markdown
Owner Author

Agreed — presence-as-a-segment doesn't prove scoping. Fixed in f9c2651.

Added a shared namespaceScopedKeyPrefix(keyFormat, namespace) helper that returns a deterministic prefix only when {{workflow.namespace}} is a complete /-delimited segment ahead of every caller-influenced field ({{pod.name}}, {{workflow.name}}, {{workflow.creationTimestamp.*}}), and null otherwise. Both paths now use it:

  • createPodLogsMinioRequestConfig fails closed unless the config keyFormat yields such a prefix.
  • getPodLogsMinioRequestConfigfromWorkflow now receives the trusted operator keyFormat (from ARGO_KEYFORMAT, not the tenant-influenced per-namespace configmap) and requires the workflow-recorded key to startWith the derived <static>/<namespace>/ prefix — so a tenant-shaped artifact key that merely contains the namespace as a later/coincidental segment is rejected.

Test updates: a template placing a caller field before the namespace tag (including a trailing-segment namespace) is now rejected; added coverage for a bounded prefix segment, the workflow-status prefix match, and fail-closed when the trusted keyFormat isn't namespace-scoped. 22 tests pass; tsc --noEmit and Prettier clean.

Note on the workflow-status path: it now validates against the operator's global ARGO_KEYFORMAT. Deployments whose per-namespace archive layout diverges from the global keyFormat will fail closed on that path (the configured archive fallback still applies) — a deliberate, security-correct tradeoff, flagged for maintainer input.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: APPROVE

Changed files:

frontend/server/handlers/pod-logs.ts
frontend/server/workflow-helper.test.ts
frontend/server/workflow-helper.ts

Review result:

No automated findings were found in the changed code.

Residual risks: the new namespace-scoping checks are security-sensitive and depend on the configured keyFormat accurately representing the archive layout; deployments with custom artifact repository behavior should be validated in integration tests to ensure archived logs remain accessible only as intended.

@herikwebb
herikwebb force-pushed the master branch 2 times, most recently from 8e96fb2 to a68c3e7 Compare July 26, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant