fix(deploy): resolve inference coordinator host to bindable address - #743
Open
pruprakash wants to merge 3 commits into
Open
fix(deploy): resolve inference coordinator host to bindable address#743pruprakash wants to merge 3 commits into
pruprakash wants to merge 3 commits into
Conversation
create_mcore_engine passed os.environ.get("MASTER_ADDR") straight through to
MegatronLLM. When the variable is unset that is None, which is indistinguishable
from passing nothing, so MCore falls back to socket.gethostname(). Inside Docker
that resolves to the container ID, which is not a bindable interface, and the
data-parallel inference coordinator dies with:
zmq.error.ZMQError: No such device (addr='tcp://<container-id>:*')
The Ray path never hit this because megatronllm_deployable_ray sets MASTER_ADDR
to the node IP itself; the PyTriton path sets nothing, so every in-framework
Triton deploy failed on single-node Docker.
Fall back to 127.0.0.1 only when MASTER_ADDR is absent, so multi-node launches
that already export a routable address are unaffected.
Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
Defaulting to 127.0.0.1 fixed the reported Docker failure but is bindable only
locally, so a multi-node launch that did not export MASTER_ADDR would bind a
coordinator nobody else can reach - trading a loud ZMQError for a silent hang.
Resolve the hostname to its IP instead. That is what the Ray path effectively
gets by exporting the node IP as MASTER_ADDR, and it is both bindable and
reachable from other containers. Loopback remains only as a last resort if
resolution itself fails.
Measured in nvcr.io/nvidian/nemo:26.08.rc9:
socket.gethostname() -> f411026bc80c ZMQ bind FAILS
socket.gethostbyname(gethostname()) -> 172.17.0.2 ZMQ bind OK
127.0.0.1 ZMQ bind OK, not routable
Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
pruprakash
added
LLM
tests
qa_rcca_done
r0.7.0
The fabricated container ID was 12 hex characters, which detect-secrets flags as a Hex High Entropy String and fails the secrets-detector job. The test only needs a value that is not an address, so name it that rather than allowlisting a non-secret or regenerating the shared baseline. Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
Author
|
/ok to test 49943f0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes NVBug 6457372: in-framework PyTriton deploys fail on single-node Docker because
create_mcore_engine()passesos.environ.get("MASTER_ADDR")(None when unset) ascoordinator_host, so MCore falls back tosocket.gethostname()— a name, not an address —which inside Docker is the container ID and cannot be bound:
zmq.error.ZMQError: No such device (addr='tcp://f3879bac406e:*'). Present since 26.08.rc1and still reproducing on 26.08.rc9. The Ray path is unaffected because
megatronllm_deployable_ray.py:72exports the node IP asMASTER_ADDRitself; the PyTritondeployable exports nothing, so PR #719 added the parameter but no usable default.
Changes
nemo_deploy/llm/inference/inference_base.py: add_default_coordinator_host(), resolvingthe hostname to its IP and falling back to loopback only if resolution fails; use it when
MASTER_ADDRis unset.tests/unit_tests/deploy/test_inference_base.py: three tests asserting the resolved addressis used and the bare hostname never passed through, that an explicit
MASTER_ADDRstill wins,and that unresolvable hostnames fall back to loopback rather than
None.Verification
verified in
nvcr.io/nvidian/nemo:26.08.rc9.ruff checkandruff format --checkclean.nvcr.io/nvidian/nemo:26.08.rc9:socket.gethostname()→f411026bc80c(ZMQ bind FAILS);socket.gethostbyname(gethostname())→172.17.0.2(bind OK, routable);
127.0.0.1(bind OK, not routable).test_inframework_mlm_tritonandtest_inframework_mlm_llm_tritonboth FAIL on unfixedsource and both PASS against this branch on
nvcr.io/nvidian/nemo:26.08.rc9, withMASTER_ADDRunset and zero ZMQ errors (run 20260812T180350Z-77c29fc6).MASTER_ADDR=127.0.0.1against unfixed source also turns both green,confirming the mechanism independently of the code change.
Reference