‘pynvml.NVMLError_NotSupported: Not Supported’ error on DGX Spark

[LisaLiNVIDIA]

We are building Cosmos3 arm container on DGX Spark,but we ran into the issue as below:

torchrun --nproc_per_node=1 -m cosmos3.scripts.inference --checkpoint-path Cosmos3-Nano -i /workspace/inputs/omni/action_forward_dynamics_camera.json -o outputs/omni --seed=0 --warmup 2 --num-iterations 3 --resolution 256 --no-guardrails --benchmark --use-torch-compile --no-use-cuda-graphs
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/cosmos3/scripts/inference.py", line 99, in <module>
    main()
    ~~~~^^
  File "/workspace/cosmos3/scripts/inference.py", line 95, in main
    inference(args)
    ~~~~~~~~~^^^^^^
  File "/workspace/cosmos3/scripts/inference.py", line 52, in inference
    setup_args = args.setup.build_setup()
  File "/workspace/cosmos3/args.py", line 1020, in build_setup
    self._build_parallelism(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        world_size=world_size, local_world_size=local_world_size, device_memory_bytes=device_memory_bytes
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/workspace/cosmos3/args.py", line 1007, in _build_parallelism
    device_memory_bytes = _get_device_memory_bytes()
  File "/workspace/cosmos3/args.py", line 1036, in _get_device_memory_bytes
    info = pynvml.nvmlDeviceGetMemoryInfo(handle)
  File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 3762, in nvmlDeviceGetMemoryInfo
    _nvmlCheckReturn(ret)
    ~~~~~~~~~~~~~~~~^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 1083, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_NotSupported: Not Supported
E0602 11:14:24.331000 45 torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 67) of binary: /workspace/.venv/bin/python
E0602 11:14:24.331000 45 torch/distributed/elastic/multiprocessing/errors/error_handler.py:145] no error file defined for parent, to copy child error file (/tmp/torchelastic_cqswkwbj/none_9sniorw6/attempt_0/0/error.json)
Traceback (most recent call last):
  File "/workspace/.venv/bin/torchrun", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
    return f(*args, **kwargs)
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 991, in main
    run(args)
    ~~~^^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 982, in run
    elastic_launch(
    ~~~~~~~~~~~~~~~
        config=config,
        ~~~~~~~~~~~~~~
        entrypoint=cmd,
        ~~~~~~~~~~~~~~~
    )(*cmd_args)
    ~^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
    raise ChildFailedError(
    ...<2 lines>...
    )
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
cosmos3.scripts.inference FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-06-02_11:14:23
  host      : be677ad18848
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 67)
  error_file: /tmp/torchelastic_cqswkwbj/none_9sniorw6/attempt_0/0/error.json
  traceback : NoneType: None

DGX Spark:
DGX-SPARK FASTOS 1.135.29 ARM64
Driver:

root@be677ad18848:/workspace# nvidia-smi
Tue Jun  2 11:16:16 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.142                Driver Version: 580.142        CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0  On |                  N/A |
| N/A   42C    P0             10W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found

[tbraun96]

Disclosure: I work on Atlas, answering generically here since this is a GB10 platform footgun rather than a Cosmos-specific bug. We tripped on the same pynvml.NVMLError_NotSupported on DGX Spark and the fix is small.

The cause is that GB10 uses a unified-memory layout and does not expose the legacy v1 NVML memory-info namespace, so a call to nvmlDeviceGetMemoryInfo(handle) returns NVML_ERROR_NOT_SUPPORTED on Spark while it works fine on Hopper/Ada. Anything probing GPU memory through the v1 call (directly, or via a torch/device util underneath) will hit this at startup. The v2 variant works on GB10, so the durable fix is to try v2 first and fall back:

try:
    info = pynvml.nvmlDeviceGetMemoryInfo_v2(handle)
except (pynvml.NVMLError_NotSupported, AttributeError):
    info = pynvml.nvmlDeviceGetMemoryInfo(handle)

The AttributeError arm matters because older pynvml builds do not expose the _v2 symbol at all, so on those you still want the v1 path rather than a crash. This covers GB10 and keeps Hopper/Ada working. Xinference and HAMi both had the identical root cause (HAMi backported a fix in Project-HAMi/HAMi#1841), so it is a well-trodden GB10 papercut at this point.

@LisaLiNVIDIA is the failing call inside Cosmos3 a direct pynvml memory-info probe, or is it coming through a torch device-utils helper? If it is a direct pynvml call in the cosmos3 inference path, the v2-with-fallback above is a near-trivial patch, happy to point at the exact spot if you can share the traceback line.

[LisaLiNVIDIA]

We ere building our container, hope this error message helped:

torchrun --nproc_per_node=1 -m cosmos3.scripts.inference --checkpoint-path Cosmos3-Nano -i /workspace/inputs/omni/action_forward_dynamics_camera.json -o outputs/omni --seed=0 --warmup 2 --num-iterations 3 --resolution 256 --no-guardrails --benchmark --use-torch-compile --no-use-cuda-graphs
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/cosmos3/scripts/inference.py", line 99, in <module>
    main()
    ~~~~^^
  File "/workspace/cosmos3/scripts/inference.py", line 95, in main
    inference(args)
    ~~~~~~~~~^^^^^^
  File "/workspace/cosmos3/scripts/inference.py", line 52, in inference
    setup_args = args.setup.build_setup()
  File "/workspace/cosmos3/args.py", line 1020, in build_setup
    self._build_parallelism(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        world_size=world_size, local_world_size=local_world_size, device_memory_bytes=device_memory_bytes
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/workspace/cosmos3/args.py", line 1007, in _build_parallelism
    device_memory_bytes = _get_device_memory_bytes()
  File "/workspace/cosmos3/args.py", line 1036, in _get_device_memory_bytes
    info = pynvml.nvmlDeviceGetMemoryInfo(handle)
  File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 3762, in nvmlDeviceGetMemoryInfo
    _nvmlCheckReturn(ret)
    ~~~~~~~~~~~~~~~~^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 1083, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_NotSupported: Not Supported
E0602 11:14:24.331000 45 torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 67) of binary: /workspace/.venv/bin/python
E0602 11:14:24.331000 45 torch/distributed/elastic/multiprocessing/errors/error_handler.py:145] no error file defined for parent, to copy child error file (/tmp/torchelastic_cqswkwbj/none_9sniorw6/attempt_0/0/error.json)
Traceback (most recent call last):
  File "/workspace/.venv/bin/torchrun", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
    return f(*args, **kwargs)
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 991, in main
    run(args)
    ~~~^^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 982, in run
    elastic_launch(
    ~~~~~~~~~~~~~~~
        config=config,
        ~~~~~~~~~~~~~~
        entrypoint=cmd,
        ~~~~~~~~~~~~~~~
    )(*cmd_args)
    ~^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
    raise ChildFailedError(
    ...<2 lines>...
    )
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
cosmos3.scripts.inference FAILED

[lfengad]

NVIDIA/cosmos-framework#22 this could be related.

[lfengad]

@LisaLiNVIDIA Is this issue fixed with the above pr?

[LisaLiNVIDIA]

Hi @lfengad ，how to Verify it？Could you please help to share the verification steps?

[lfengad]

@LisaLiNVIDIA You can use the latest version comos-framework to rerun it?

[Rohithmatham12]

I opened a focused framework PR for the direct traceback path here: NVIDIA/cosmos-framework#41.

The change makes Cosmos3 inference setup try pynvml.nvmlDeviceGetMemoryInfo_v2() before the legacy v1 memory-info API, keeps the v1 fallback for older pynvml builds, and retains the existing torch/default fallback. This should cover DGX Spark / GB10 systems where the legacy call raises NVMLError_NotSupported.

[LisaLiNVIDIA]

Tried with the cosmos3-internal repo

root@231f444b98d0:/workspace# torchrun --nproc_per_node=1 -m cosmos3.scripts.inference --checkpoint-path Cosmos3-Nano -i /workspace/inputs/omni/t2v.json -o outputs/omni --seed=0 --resolution 256 --no-guardrails --benchmark --use-torch-compile
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/workspace/cosmos3/scripts/inference.py", line 99, in <module>
main()
~~~~^^
File "/workspace/cosmos3/scripts/inference.py", line 95, in main
inference(args)
~~~~~~~~~^^^^^^
File "/workspace/cosmos3/scripts/inference.py", line 52, in inference
setup_args = args.setup.build_setup()
File "/workspace/cosmos3/args.py", line 1020, in build_setup
self._build_parallelism(
~~~~~~~~~~~~~~~~~~~~~~~^
world_size=world_size, local_world_size=local_world_size, device_memory_bytes=device_memory_bytes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/workspace/cosmos3/args.py", line 1007, in _build_parallelism
device_memory_bytes = _get_device_memory_bytes()
File "/workspace/cosmos3/args.py", line 1036, in _get_device_memory_bytes
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 3762, in nvmlDeviceGetMemoryInfo
_nvmlCheckReturn(ret)
~~~~~~~~~~~~~~~~^^^^^
File "/workspace/.venv/lib/python3.13/site-packages/pynvml.py", line 1083, in _nvmlCheckReturn
raise NVMLError(ret)
pynvml.NVMLError_NotSupported: Not Supported
E0617 05:37:31.778000 42 torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 64) of binary: /workspace/.venv/bin/python
E0617 05:37:31.779000 42 torch/distributed/elastic/multiprocessing/errors/error_handler.py:145] no error file defined for parent, to copy child error file (/tmp/torchelastic_7sr_3vq5/none_3jsew_1j/attempt_0/0/error.json)
Traceback (most recent call last):
File "/workspace/.venv/bin/torchrun", line 10, in <module>
sys.exit(main())
~~~~^^
File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/errors/**init**.py", line 362, in wrapper
return f(*args, **kwargs)
File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 991, in main
run(args)
~~~^^^^^^
File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/run.py", line 982, in run
elastic_launch(
~~~~~~~~~~~~~~~
config=config,
~~~~~~~~~~~~~~
entrypoint=cmd,
~~~~~~~~~~~~~~~
)(*cmd_args)
~^^^^^^^^^^^
File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 170, in **call**
return launch_agent(self._config, self._entrypoint, list(args))
File "/workspace/.venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
raise ChildFailedError(
...<2 lines>...
)
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

cosmos3.scripts.inference FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time      : 2026-06-17_05:37:30
host      : 231f444b98d0
rank      : 0 (local_rank: 0)
exitcode  : 1 (pid: 64)
error_file: /tmp/torchelastic_7sr_3vq5/none_3jsew_1j/attempt_0/0/error.json
traceback : NoneType: None

============================================================

[Rohithmatham12]

A good verification path is to test with a Cosmos Framework build that includes NVIDIA/cosmos-framework#41, then rerun the same DGX Spark inference command that originally failed.\n\nSuggested steps inside the DGX Spark ARM container:\n\n1. Rebuild or reinstall Cosmos Framework from the PR branch/commit that contains the NVML v2 fallback.\n2. Optional quick NVML sanity check:\n\nbash\npython - <<'PY'\nimport pynvml\n\npynvml.nvmlInit()\ntry:\n handle = pynvml.nvmlDeviceGetHandleByIndex(0)\n try:\n info = pynvml.nvmlDeviceGetMemoryInfo_v2(handle)\n print("NVML_V2_OK", info.total)\n except AttributeError:\n info = pynvml.nvmlDeviceGetMemoryInfo(handle)\n print("NVML_V1_OK", info.total)\nfinally:\n pynvml.nvmlShutdown()\nPY\n\n\n3. Rerun the original repro command:\n\nbash\ntorchrun --nproc_per_node=1 -m cosmos3.scripts.inference \\n --checkpoint-path Cosmos3-Nano \\n -i /workspace/inputs/omni/action_forward_dynamics_camera.json \\n -o outputs/omni \\n --seed=0 \\n --warmup 2 \\n --num-iterations 3 \\n --resolution 256 \\n --no-guardrails \\n --benchmark \\n --use-torch-compile \\n --no-use-cuda-graphs\n\n\nExpected result: startup should no longer fail at pynvml.nvmlDeviceGetMemoryInfo(handle) with NVMLError_NotSupported. On GB10/DGX Spark the code should use the v2 memory-info path; on older pynvml builds it should still fall back to the legacy v1 path.

[lfengad]

NVIDIA/cosmos-framework#41 solved this! THX for the tests and contributions!