Skip to content

fix(frontend): enforce domain allowlist on s3/minio artifact endpoint#98

Open
herikwebb wants to merge 13 commits into
masterfrom
security/fix-s3-endpoint-ssrf-20260721014523
Open

fix(frontend): enforce domain allowlist on s3/minio artifact endpoint#98
herikwebb wants to merge 13 commits into
masterfrom
security/fix-s3-endpoint-ssrf-20260721014523

Conversation

@herikwebb

Copy link
Copy Markdown
Owner

Summary

Closes the SSRF path tracked in #97. The ml-pipeline-ui /artifacts/get handler accepts an object-store endpoint via the providerInfo query parameter for source=s3 and source=minio. The http/https sources are gated by isAllowedDomain(url, ALLOWED_ARTIFACT_DOMAIN_REGEX), but the s3/minio provider endpoint was never checked, so a request could direct the server's outbound object-store fetch at an arbitrary host:port (including in-cluster services and 169.254.169.254) and have the response streamed back — a full-read server-side request forgery, unauthenticated in the default standalone deployment.

This subjects the provider-supplied endpoint to the same isAllowedDomain check already applied to http/https sources, rejecting endpoints outside the configured allowlist with 400 before any client is constructed.

Why this is safe / minimal

Testing

  • New regression test asserts a restricted allowlist rejects an attacker-chosen s3 endpoint and never constructs the client.
  • frontend/server: tsc --noEmit clean, Prettier clean, artifact-get/artifacts/artifact-auth suites pass (79 tests).

Follow-up

Per kubeflow/pipelines SECURITY.md, upstream disclosure should be coordinated privately (GitHub Security Advisory on kubeflow/pipelines or ksc@kubeflow.org). This PR targets the fork's synced master; once reviewed, the change is intended to be promoted upstream via that private channel.

Tracking issue: #97


Generated by Claude Code

Herik Webb and others added 12 commits July 21, 2026 01:44
Signed-off-by: Herik Webb <herik@shardplate.local>
Signed-off-by: Herik Webb <herik@shardplate.local>
Add a pull request watch policy so that any agent (including scheduled
routines) subscribes to a PR immediately after opening it, handing it to
the persistent watcher for review-feedback and CI follow-up until the PR
is merged or closed.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
…tter

Codify that every commit must be authored, committed, and signed off by the
human submitter rather than an agent identity. Automated environments default
the git identity to the agent (e.g. Claude <noreply@anthropic.com>), so a bare
'git commit -s' signs off as the agent and breaks the DCO requirement. Document
setting the repo-local identity to the submitter, verifying author and sign-off
before pushing, and reusing the submitter identity for follow-up commits on a
PR they already authored.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
gpt-5.5 is a reasoning model whose reasoning tokens count against
max_output_tokens, so the previous 3000 budget was intermittently exhausted
before any review text was produced, failing the review job with 'OpenAI API
response did not include review text'. Raise the default to 16000 and make it
overridable via the OPENAI_MAX_OUTPUT_TOKENS repository variable.

Also export the variable in review-pr.sh so the python child process actually
inherits it — previously it was set without export, so the child always used
its own hardcoded default regardless of the shell value.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Raising the token budget means a review can exceed GitHub's 65536-character
comment body limit, which would make 'gh pr comment' fail after a successful
model call. Truncate the body (with margin) and append a notice when it is too
large instead of failing to post the review.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
`resolveFilePathOnVolume` in `frontend/server/utils.ts` composed the final
filesystem path with `path.join(volumeMountPath, filePathInVolume)`, which
collapses `..` segments. `getArtifactsHandler` in
`frontend/server/handlers/artifacts.ts` validates the `bucket` query param
with `isAllowedResourceName` but only length-caps the `key` param and hands
it to `getVolumeArtifactsHandler` → `findFileOnPodVolume` →
`resolveFilePathOnVolume` unchanged.

Combined with a well-known volume name mounted on the ml-pipeline-ui pod
(for example `config-volume` from
`manifests/kustomize/base/pipeline/ml-pipeline-ui-deployment.yaml`), an
authenticated Kubeflow user could pass
`?source=volume&bucket=config-volume&key=../../var/run/secrets/kubernetes.io/serviceaccount/token`
through the artifacts auth middleware — which only checks that they can
access their own namespace — and stream the ml-pipeline-ui pod's service
account token (or any other file the pod's filesystem contains).

Normalize the joined path with `path.resolve` and require that it stays
under the resolved `volumeMountPath` (equal to it or under
`volumeMountPath + '/'`). This blocks both `../` escapes when no `subPath`
is set and `subPath + '/../../'` escapes when `subPath` is configured. Add
two `resolveFilePathOnVolume` regression tests covering both variants.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
…eads

The lexical path.resolve containment check in resolveFilePathOnVolume stops
'..' traversal but not a symlink inside the mounted volume pointing outside it
(e.g. 'link -> /etc/passwd'), which fs.createReadStream would follow at read
time.

Add resolveRealPathWithinMount, which resolves the real (symlink-free) path
immediately before the read and re-verifies it stays within the volume mount,
minimizing the TOCTOU window by returning the fully resolved path to open.
findFileOnPodVolume now also returns the resolved volume mount path so the
read site can perform this check; the extra tuple element is backward
compatible with existing two-element destructures. getVolumeArtifactsHandler
calls the resolver before stat/createReadStream and reads the returned safe
path. When the path cannot be resolved (missing file) the original path is
returned so existing not-found behavior is preserved.

Extract a shared isPathInsideMount helper used by both the lexical and the
realpath containment checks. Add resolveRealPathWithinMount tests covering an
in-mount file, a symlink escaping the mount, and an unresolvable path.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
The previous fix resolved the real path and re-checked containment, but the
handler then reopened that path with stat()/createReadStream(). An attacker
able to write the volume could swap the file for a symlink between the check
and the open, and both calls follow symlinks, so the read could still escape.

Replace resolveRealPathWithinMount with openRealFileWithinMount, which opens
the file once, verifies the opened descriptor's real target (via
/proc/self/fd on Linux, falling back to path resolution elsewhere) is inside
the mount, and returns the handle. getVolumeArtifactsHandler streams from that
same handle instead of reopening by name, so validation and read act on one fd
and cannot be raced. Open errors (missing file) still propagate to preserve the
existing not-found behavior.

Update tests to cover the open-and-validate handle, symlink rejection, and
missing-file propagation.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Match the notebook-intelligence PR reviewer:
- Trigger on pull_request instead of pull_request_target and check out the PR
  head directly (fork PRs get a read-only token and no secrets, so running the
  diff-only reviewer on the checked-out head is safe and simpler).
- Emit a machine-readable VERDICT (APPROVE / CHANGES_REQUESTED) and fail the
  check on a non-APPROVE verdict, so a green 'review' check means the reviewer
  signed off. This is what an automated promotion can key off.
- Add an 'override-review-gate' label escape hatch that relaxes only the exit
  status; the review still runs and comments.
- Use a three-dot (merge-base) diff range with a two-dot fallback on shallow
  clones, so the reviewer sees what the PR introduces on merge.

Keeps the fork's OPENAI_MAX_OUTPUT_TOKENS=16000 override: gpt-5.5 is a reasoning
model and 3000 was easily exhausted, producing empty responses that would now
fail the review gate.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
Add a workflow_dispatch action + scripts/promote-upstream.sh that opens a
cross-fork PR from a chosen fork branch into kubeflow/pipelines, using a
human PAT (secrets.UPSTREAM_PAT) since the default GITHUB_TOKEN cannot write
upstream. It resolves the upstream default branch, refuses empty promotions,
guards against leaking fork-only automation files into the upstream diff, and
by default requires the branch's fork PR review gate to be green before
promoting (override with require_review_pass=false). Change detection uses
GitHub's server-side compare API so it is independent of local clone depth.

Signed-off-by: Herik Webb <herikwebb@users.noreply.github.com>
The ml-pipeline-ui artifact endpoint accepts an object-store endpoint via
the `providerInfo` query parameter. The http/https artifact sources are
constrained by the ALLOWED_ARTIFACT_DOMAIN_REGEX allowlist, but the
s3/minio provider endpoint was never checked, so a request could direct
the server's outbound object-store fetch at an arbitrary host and port and
have the response streamed back — a server-side request forgery reachable
without authentication in the default standalone deployment.

Subject the provider-supplied endpoint to the same isAllowedDomain check
already applied to http/https sources, rejecting endpoints outside the
configured allowlist before any client is constructed. The default regex
remains permissive, so existing deployments are unaffected; operators who
restrict the allowlist are now protected consistently across all artifact
sources. Adds a regression test covering the rejected case.
@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: APPROVE

Changed files:

frontend/server/handlers/artifacts.ts
frontend/server/integration-tests/artifact-get.test.ts

Review result:

No automated findings were found for the changed code.

Residual risks: the effectiveness of the new SSRF guard still depends on the behavior and operator configuration of isAllowedDomain / ALLOWED_ARTIFACT_DOMAIN_REGEX, and the added test covers the S3 rejection path but not the MinIO source or an allowed endpoint under a restrictive allowlist.

… SSRF guard

Extend the artifact endpoint allowlist regression tests per review feedback:
add a minio-source rejection case (the guard covers source=minio as well as
s3), and an allowed-endpoint case asserting a restrictive allowlist still
permits the operator's own object store and constructs the client. Closes the
coverage gap noted in the automated review of the s3/minio SSRF fix.

Copy link
Copy Markdown
Owner Author

Thanks for the review. Addressed the test-coverage residual risk in e37831f: added a source=minio rejection case and an allowed-endpoint case (a restrictive ALLOWED_ARTIFACT_DOMAIN_REGEX still permits the operator's own object store and constructs the client). Artifact-get suite now at 43 passing tests; Prettier/tsc clean.

The other residual note — effectiveness depending on isAllowedDomain/ALLOWED_ARTIFACT_DOMAIN_REGEX and operator configuration — is by design: this PR makes the s3/minio path honor the same allowlist knob that already gates http/https, rather than silently bypassing it. Hardening the permissive default (^.*$) is tracked separately in #89 so it applies uniformly across all sources.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

Automated PR Review

Model: gpt-5.5
Verdict: APPROVE

Changed files:

frontend/server/handlers/artifacts.ts
frontend/server/integration-tests/artifact-get.test.ts

Review result:

No automated findings were found in the changed code.

Residual risks: the effectiveness of the SSRF guard depends on the existing isAllowedDomain URL/regex semantics and on operators configuring a restrictive ALLOWED_ARTIFACT_DOMAIN_REGEX; DNS rebinding or overly broad regexes remain operational risks.

@herikwebb
herikwebb force-pushed the master branch 5 times, most recently from 8e96fb2 to a68c3e7 Compare July 26, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant