lazy-load model handlers to decouple megatron deps from tinker path#700
Open
hardik-vala wants to merge 1 commit into
Open
lazy-load model handlers to decouple megatron deps from tinker path#700hardik-vala wants to merge 1 commit into
hardik-vala wants to merge 1 commit into
Conversation
The Tinker backend's call to art.dev.get_model_config transitively imported art.megatron.model_support.registry, which eagerly imported every handler module — including handlers/qwen3_5.py, whose top-level `from megatron.core...` made `megatron-core` a hard install-time dependency for Tinker users. This change introduces a single source of truth for handler key, status, and lazy-load address as `HandlerMeta` records in spec.py. The registry builds specs from these metas and dynamically imports the corresponding handler module on the first call to get_model_support_handler*. The handler classes themselves read their `key` / `native_vllm_lora_status` from the same metas so the values can't drift. Result: `art.tinker` is importable and `model.register()` runs without megatron-core installed, while megatron-equipped paths see no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
from art.tinker import TinkerBackend; await model.register(backend)previously failed withModuleNotFoundError: No module named 'megatron'on environments withoutmegatron-coreinstalled (e.g. macOS, where the[megatron]extra can't install due totransformer-engine/apex/deep-ep).tinker._get_service → art.dev.get_model_config → art.megatron.model_support.registry, whose top-level handler imports pulledhandlers/qwen3_5.pyand its top-levelfrom megatron.core....HandlerMetaclass inspec.pyas the single source of truth for each registered handler'skey,native_vllm_lora_status, and lazy-load address (module,attr).registry.pybuilds everyModelSupportSpecfrom these metas and resolves handlers lazily viaimportlib.import_moduleon the first call toget_model_support_handler*. The handler classes also read theirkey/native_vllm_lora_statusfrom the same metas so the values can't drift.import art.tinker,from art.dev.get_model_config import get_model_config, andTinkerBackend._get_service(model)all succeed withoutmegatron-core. Megatron-equipped paths are unchanged — same handler instances, just imported on first lookup instead of at registry load.Test plan
pip install -e '.[tinker]'and nomegatron-core:python -c "import art.tinker.backend"succeedsget_model_config(base_model='Qwen/Qwen3-4B-Instruct-2507', ...)returns the expectedpeft_args["target_modules"]default_target_modules_for_model('Qwen/Qwen3-30B-A3B-Instruct-2507')returns[..., 'experts']await TinkerBackend()._get_service(model)constructsTinkerServicewithout importingmegatronuv run prek run --all-filesclean