bug/feat: osmo dataset upload fails on S3-compatible backends that reject path-style GetBucketLocation (CAIOS, R2-strict, Wasabi-strict)

[narenandu]

Describe the bug.

The OSMO CLI (6.2.10) issues a GetBucketLocation request as a precheck on every data-plane operation (osmo dataset upload, osmo data upload, osmo data check, etc.). The bundled boto3 client is configured with the default addressing_style: path, so the precheck is sent in path-style:

GET https://<endpoint>/<bucket>/?location

S3-compatible object stores that only honor GetBucketLocation in virtual-hosted-style reject this with:

ClientError: An error occurred (PathStyleRequestNotAllowed) when calling
the GetBucketLocation operation: The path style requests are not allowed
for this method, please switch to hostname-based requests.

Because OSMO gates the rest of the upload pipeline on this precheck, the CLI cannot upload, download, or even check a bucket on these stores. As a result, the OSMO UI Datasets view stays permanently empty for these deployments — there is no path to register a dataset.

This affects any S3-compatible backend with this constraint. The case observed in the wild was CoreWeave Object Storage (CAIOS) / cwobject.com on a CoreWeave CKS deployment.

Affected versions. 6.2.10 (CLI + service charts 1.2.1). Likely earlier.

---

Reproduction

Setup: an OSMO 6.2.10 deployment with a CAIOS-backed bucket configured. The bucket itself is reachable from outside OSMO using path-style for ListObjectsV2 / PutObject / GetObject — only GetBucketLocation is virtual-host-only.

# Cluster-side: register the dataset bucket so the CLI sees it
curl -sf -X PUT -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  $OSMO_API/api/configs/dataset -d '{
    "configs": {
      "default_bucket": "default",
      "buckets": {
        "default": {
          "dataset_path": "s3://my-caios-bucket/datasets",
          "region": "us-east-1",
          "mode": "read-write",
          "default_credential": {
            "endpoint": "s3://my-caios-bucket/datasets",
            "region": "us-east-1",
            "override_url": "https://cwobject.com",
            "access_key_id": "...",
            "access_key": "..."
          }
        }
      }
    }
  }'
# HTTP 200, bucket registered

# Client-side: bucket is visible
$ osmo bucket list
Bucket              Description   Location                       Mode         Default Cred
default (default)   ...           s3://my-caios-bucket/datasets  read-write   Yes

$ osmo profile set bucket default
Profile Set

# Now any data-plane op fails identically:

$ osmo dataset upload smoke:v1 ./small-dir --desc "smoke"
Unknown error: An error occurred (PathStyleRequestNotAllowed) when calling
  the GetBucketLocation operation: The path style requests are not allowed
  for this method, please switch to hostname-based requests.
Error code: 1

$ osmo data upload s3://my-caios-bucket/datasets/foo ./small-dir
Unknown error: An error occurred (PathStyleRequestNotAllowed) ...

$ osmo data check s3://my-caios-bucket/datasets
Unknown error: An error occurred (PathStyleRequestNotAllowed) ...

The standard env-var workaround for boto3 does NOT bite, because boto3.client('s3', config=Config(s3={'addressing_style': 'path'})) overrides the env:

$ AWS_S3_ADDRESSING_STYLE=virtual osmo dataset upload smoke:v1 ./small-dir
Unknown error: An error occurred (PathStyleRequestNotAllowed) ...

Setting the dataset's default_credential.override_url to a virtual-hosted-style URL makes the SDK double-prepend the bucket and the URL no longer resolves:

# override_url = https://my-caios-bucket.cwobject.com
$ aws --endpoint-url https://my-caios-bucket.cwobject.com s3 ls s3://my-caios-bucket/
Could not connect to the endpoint URL:
"https://my-caios-bucket.my-caios-bucket.cwobject.com/?list-type=2&..."

Direct aws s3 cp (without OSMO) works fine against https://cwobject.com in path-style for PutObject / ListObjects / GetObject. So the bucket and credentials are correct; the CLI's GetBucketLocation precheck is the only step that fails.

---

Why this matters

1. The OSMO UI Datasets view is empty by design when running on CAIOS — there is no CLI/UI path to register a dataset. Workflows can still submit, schedule, and complete (the in-cluster credential paths are unaffected), but datasets / app pushes / direct data ops are blocked.
2. Affects every S3-compatible store with this constraint. CAIOS surfaces this on a stock CoreWeave CKS install of OSMO. Other tenants on Cloudflare R2 / Backblaze B2 / Wasabi with similar path-style policies would hit the same issue.
3. No graceful surface. The error bubbles up as a generic Unknown error: An error occurred (PathStyleRequestNotAllowed). There is no hint that the underlying problem is "your S3 backend doesn't support path-style GetBucketLocation, configure addressing-style virtual" — operators end up debugging boto3 internals.
4. Bucket region is already known. The BucketConfig.region field is mandatory (default: us-east-1), so the GetBucketLocation precheck is computing something OSMO already has.

---

Suggested fix

Three concrete options, ordered by impact and minimum scope:

Option A — drop the GetBucketLocation precheck. BucketConfig.region is already mandatory in OSMO's data model. Whatever the precheck is using the location for can be sourced from there directly. This is the minimal-diff fix and unblocks every backend that doesn't speak path-style GetBucketLocation.

Option B — add addressing_style to BucketConfig (and propagate to the boto3 Config). Per-bucket toggle. Default "auto" (current behavior); operators set "virtual" for CAIOS / R2 / etc. Backwards compatible.

Option C — honor AWS_S3_ADDRESSING_STYLE in the boto3 client init. Don't pass an explicit addressing_style in the Config so the SDK falls back to env / ~/.aws/config. Lowest-friction for operators who already use the standard AWS env.

Probably (A) + (C) together: drop the unnecessary precheck AND stop hardcoding addressing-style. (B) becomes optional polish.

The relevant code is the boto3 client factory used by osmo dataset upload / osmo data upload / osmo data check. From the CLI bundle structure (/usr/local/osmo/osmo-cli/_internal/), it appears OSMO modules are compiled into the PyInstaller PYZ, so I haven't been able to point at the exact line — but the call site is whichever helper builds the boto3 s3 client for these subcommands.

---

Documentation gap (separate, smaller fix)

The OSMO deployment guide should call out CAIOS / non-AWS S3 backends as a known incompatibility for the dataset CLI in 6.2.10, with the upstream issue linked. Today the failure is opaque from the user's side — the error doesn't suggest "this is a known incompatibility, here's the workaround".

---

Workaround

Until upstream lands a fix, the only way to populate a CAIOS-backed bucket on OSMO is direct aws s3 cp (path-style works for PutObject / ListObjects / GetObject):

AWS_ACCESS_KEY_ID=$AKID AWS_SECRET_ACCESS_KEY=$SAK \
  aws --endpoint-url https://cwobject.com --region us-east-1 \
  s3 cp ./local-file s3://my-caios-bucket/path/

This bypasses OSMO entirely — files land in the bucket but are NOT registered as OSMO datasets, so the UI Datasets view stays empty.

---

Environment

• OSMO 6.2.10 (charts 1.2.1 / app 6.2.10.fa46f7f09)
• OSMO CLI installed via the published Mac binary
• Kubernetes v1.35.4 on CoreWeave CKS
• Backend: CoreWeave Object Storage (https://cwobject.com) — S3-compatible
• Reproduced on CKS cluster after fresh deploy of OSMO + a CAIOS bucket configured per the deployment guide.

Cross-references:

• bug/feat: BACKEND scheduler_type enum hardcoded to 'kai' — no fallback to K8s default-scheduler in 6.2.10 #936 — BACKEND.scheduler_settings.scheduler_type enum hardcoded to 'kai'. Same flavor of "data model exposes a setting that the runtime treats as a single-valued switch", different field.

[narenandu]

Sibling failure mode — in-cluster log archival to S3 also broken on CAIOS

Filing under the same issue since the root cause looks identical, just on a different boto3 op and a different code path. Happy to split into a separate issue if maintainers prefer.

Symptom. On the same 6.2.10 deployment against CAIOS, completed workflows have no historical log archive in the configured workflow_log S3 bucket. The OSMO UI / osmo workflow logs <id> returns the live-tailed buffer while the workflow's pods exist (osmo-ctrl WebSocket → osmo-logger), but once the pod is gone the API has no S3 fallback — the task_logs_<task>_<retry>.txt blob OSMO expects there was never written. This is meaningful regression: it removes the platform's ability to retain historical logs for audit / post-mortem / re-review of completed runs, which is one of the primary reasons to wire an object store under OSMO in the first place.

Evidence. Two consecutive isaaclab-train-* workflows on a fresh OSMO 6.2.10 install:

$ osmo workflow query isaaclab-train-3 -t json | jq '.status, .end_time'
"COMPLETED"
"2026-05-01T22:59:11.813054"

$ AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… \
    aws s3 ls --endpoint-url https://cwobject.com \
    s3://omniverse-poc-cat/workflows/isaaclab-train-3/
# (no output — prefix does not exist)

The bucket itself works for some in-cluster code paths — e.g. data-pass-via-s3-* test workflows successfully wrote payload.txt (~363 B) using PutObject in path-style on the same credentials/endpoint. So PutObject against CAIOS in path-style is fine; the failing op is something earlier in the pipeline.

osmo-service stack trace when the API tries to read the (never-written) archived log for the prior multi-task run isaaclab-train-2:

2026-05-01T22:51:50.455+00:00 service [WARNING] s3: No entries found for prefix:
    workflows/isaaclab-train-2/task_logs_master_0.txt
2026-05-01T22:51:50.456+00:00 service [ERROR] service: Got an exception of type
    OSMODataStorageClientError on url path /api/workflow/isaaclab-train-2/logs

botocore.exceptions.ClientError: An error occurred (400) when calling the
    HeadObject operation: Bad Request

  File "/osmo_workspace+/src/service/core/service_binary.runfiles/
        osmo_workspace+/src/lib/data/storage/backends/s3.py", line 281, in handle_error
    raise client.OSMODataStorageClientError(...)

Repeats for ~30 seconds (HeadObject retried three times per request, request keeps polling).

Mapping to #940's analysis.

Aspect	#940 (CLI dataset/data ops)	This (in-cluster log archival)
Caller	OSMO CLI on user host	osmo-service (src/lib/data/storage/backends/s3.py:281); likely osmo-ctrl on the write side
boto3 op	GetBucketLocation	HeadObject (observed read failure); PutObject precheck presumed for write
Error from CAIOS	PathStyleRequestNotAllowed	400 Bad Request (less specific, same family)
Net effect	UI Datasets view permanently empty	UI / API logs view empty for any completed workflow once the pod is gone

The two errors look superficially different (PathStyleRequestNotAllowed vs 400 Bad Request) but the boto3 client construction is the same family of bug — an op that is hardcoded to path-style addressing and fails against an S3-compatible store that requires virtual-hosted style for that specific op. CAIOS happens to surface a generic 400 for HeadObject rather than the structured PathStyleRequestNotAllowed it returns for GetBucketLocation, so it's worth maintainers checking the request signing on HeadObject to confirm.

Suggested fix scope. The fix proposed in #940 (drop unnecessary GetBucketLocation precheck, expose addressing_style on BucketConfig, honor AWS_S3_ADDRESSING_STYLE) needs to be applied in both the CLI module and the service binary's S3 backend (src/lib/data/storage/backends/s3.py). A CLI-only fix would still leave OSMO without historical log archival on CAIOS. Option (A) — drop the precheck — is particularly attractive here since the bucket region is already mandatory in BucketConfig.

Why this matters beyond CLI ergonomics. Once osmo-logger's in-pod EmptyDir / Postgres buffer turns over (pod restart, retention window), there is no surviving copy of the task logs anywhere. For training workflows in particular this is a hard regression — operators have no way to come back days later and review what happened on a completed run. Object-store archival is the platform feature OSMO already exposes through the workflow_log credential; it just doesn't actually write to it on CAIOS today.

Environment. Same as #940. CoreWeave CKS, OSMO 6.2.10 / charts 1.2.1, CAIOS (https://cwobject.com).

[narenandu]

This is a BLOCKER for using OSMO on CoreWeave with the CW AI Object Storage

There is a similar long-standing PR in s5cmd repo, that adds the support for virtual host style endpoints. We can use this as reference if needed to update the code. There is even a consolidated PR someone created, that has the fix (not needed, but FYI)

Sharing the fix here that we implemented for s5cmd upstream repo forked with in CoreWeave meanwhile !

[fernandol-nvidia]

Hi @narenandu,

Is AWS_S3_ADDRESSING_STYLE an official botocore recognized environment variable?

From OSMO's end, we are not overriding any addressing_style (https://github.com/NVIDIA/OSMO/blob/main/src/lib/data/storage/backends/s3.py#L111) before instantiating a boto3 client.

Tangentially, bypassing region check makes sense and that's an optimization applicable to Dataset. But I think we should ultimately fix the addressing style regardless of the GetBucketLocation call.

AWS S3 recommends setting addressing_style via a config (https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#addressing-style), for example:

# ~/.aws/config
[default]
s3 =
    addressing_style = virtual

Is this something you can try?

[narenandu]

I pretty much have the addressing_style in my config ! and I use my default profile. May be I need to spot the location of the boto configuration that OSMO uses. Let me check

cat ~/.aws/config
[default]
endpoint_url = https://cwobject.com
s3 =
   addressing_style = virtual

[cypres]

Verified that #950 fixes the dataset cli commands

osmo config show DATASET
{
  "buckets": {
    "caios": {
      "dataset_path": "s3://osmo-on-cw-dev-harnholm-datasets",
      "region": "us-east-14a",
      "description": "",
      "mode": "read-write",
      "default_credential": {
        "endpoint": "s3://osmo-on-cw-dev-harnholm-datasets/datasets",
        "region": "us-east-14a",
        "override_url": "https://cwobject.com",
        "addressing_style": null,
        "access_key_id": "*********",
        "access_key": "**********"
      }
    }
  },
  "default_bucket": "caios"
}

osmo dataset upload test:v1 (pwd)/lipsum --desc "test"
Uploading to Dataset test version 1 bucket caios
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.49k/3.49k [00:01<00:00, 2.76kB/s, file_name=/Users/harnholm/code/osmo/lipsum/lipsum.txt]
Writing manifest file...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1927.53it/s]
Dataset test has been uploaded to bucket caios
Uploaded 3.5 KiB new data to the storage.

osmo data list s3://osmo-on-cw-dev-harnholm-datasets/
eRrKJmsdTWqLUY2ZtVOdHw/

set -l awscfg (mktemp -d)/config
printf '[default]\ns3 =\n    addressing_style = virtual\n' > $awscfg
env AWS_CONFIG_FILE=$awscfg AWS_ACCESS_KEY_ID=**** AWS_SECRET_ACCESS_KEY=*** aws --endpoint-url https://cwobject.com s3 ls s3://osmo-on-cw-dev-harnholm-datasets/ --recursive
2026-05-05 09:32:06       3569 eRrKJmsdTWqLUY2ZtVOdHw/hashes/e992bc5cb16baf3af7d92ad9565cabae
2026-05-05 09:32:06        383 eRrKJmsdTWqLUY2ZtVOdHw/manifests/1.json

env AWS_CONFIG_FILE=$awscfg AWS_ACCESS_KEY_ID=**** AWS_SECRET_ACCESS_KEY=***** aws --endpoint-url https://cwobject.com s3 cp s3://osmo-on-cw-dev-harnholm-datasets/eRrKJmsdTWqLUY2ZtVOdHw/hashes/e992bc5cb16baf3af7d92ad9565cabae -
Lorem ipsum dolor sit amet, consectetur adipiscing elit...

[cypres]

Validated for workflow logs as well

osmo config show WORKFLOW
{
  "workflow_data": {
    "credential": {
      "endpoint": "s3://osmo-on-cw-dev-harnholm-datasets/workflow-data",
      "region": "us-east-14a",
      "override_url": "https://cwobject.com",
      "addressing_style": null,
      "access_key_id": "**********",
      "access_key": "**********"
    },
    "base_url": "",
    "websocket_timeout": 1440,
    "data_timeout": 10,
    "download_type": "download"
  },
  "workflow_log": {
    "credential": {
      "endpoint": "s3://osmo-on-cw-dev-harnholm-datasets/workflow-log",
      "region": "us-east-14a",
      "override_url": "https://cwobject.com",
      "addressing_style": null,
      "access_key_id": "**********",
      "access_key": "**********"
    }
  },

aws --endpoint-url https://cwobject.com s3 ls s3://osmo-on-cw-dev-harnholm-datasets/ --recursive
2026-05-05 09:32:06       3569 eRrKJmsdTWqLUY2ZtVOdHw/hashes/e992bc5cb16baf3af7d92ad9565cabae
2026-05-05 09:32:06        383 eRrKJmsdTWqLUY2ZtVOdHw/manifests/1.json
2026-05-05 14:17:57       6998 workflow-log/hello-osmo-17/hello.spec
2026-05-05 14:18:14        399 workflow-log/hello-osmo-17/task_logs_hello_0.txt
2026-05-05 14:18:14        831 workflow-log/hello-osmo-17/workflow_events.txt
2026-05-05 14:18:14        399 workflow-log/hello-osmo-17/workflow_logs.txt
2026-05-05 14:17:57        381 workflow-log/hello-osmo-17/workflow_spec.yaml

[cypres]

Fixed in #950 and #957