feat: add scheduler backends (local/k8s/slurm) with unified CLI#10

Merged

TerrenceZhangX merged 56 commits intomicrosoft:mainfrom

TerrenceZhangX:zhangt/scheduler-support

Mar 20, 2026

Commits on Mar 17, 2026

feat: add K8s and Slurm scheduler backends for profiling jobs
Terrence Zhang
committed
feat: switch to proper API clients for remote submission
Terrence Zhang
committed
chore: add proper pyproject.toml with dependency declarations
Terrence Zhang
committed
fix: make pip install -e . and entry point actually work
Terrence Zhang
committed
refactor: unified CLI as 'flowsim submit' instead of 'flowsim-submit'
Terrence Zhang
committed
fix: 'flowsim submit' submits by default, --dry-run to preview
Terrence Zhang
committed
fix: validate cluster connection params before submit
Terrence Zhang
committed
feat: support env vars for cluster connection params
Terrence Zhang
committed
feat: config-first approach with flowsim init
Terrence Zhang
committed
refactor: flowsim init takes CLI args instead of interactive prompts
Terrence Zhang
committed
feat: PD disaggregation support + multi-node Docker test infra
Terrence Zhang
committed
chore: add dev-setup.sh/dev-teardown.sh for one-click test clusters
Terrence Zhang
committed
feat: add local scheduler backend — flowsim submit --scheduler local
Terrence Zhang
committed
test: add 61 unit tests for scheduler CLI, backends, and config
Terrence Zhang
committed
feat: persistent logs under output_dir, flowsim status/logs, refuse K8s submit without PVC
Terrence Zhang
committed
fix: flowsim logs shows all log files (stdout + stderr) with listing
Terrence Zhang
committed
fix: flowsim logs shows file locations + actionable commands instead of dumping content
Terrence Zhang
committed
test: integration tests for all 3 scheduler backends (local/k8s/slurm)
Terrence Zhang
committed
feat: align CLI with standard job platform APIs
Terrence Zhang
committed
fix: CLI only shows scheduler-specific args based on --scheduler
Terrence Zhang
committed
fix: use python3 instead of python in profile command
Terrence Zhang
committed
fix: use YYYYMMDD_HHMMSS timestamp in log filenames
Terrence Zhang
committed

Commits on Mar 18, 2026

scheduler: add Slurm CLI mode (sbatch/squeue/scancel) + integration test
Terrence Zhang
committed
slurm: use YYYYMMDD_HHMMSS timestamp for output dirs (consistent with local/k8s)
Terrence Zhang
committed
docs: add scheduler README with CLI usage and architecture overview
Terrence Zhang
committed
profile: add --sweep for multi-point profiling in one job
Terrence Zhang
committed
test: add sweep integration tests (inline + file)
Terrence Zhang
committed
refactor: dedup shared utilities, deprecate Slurm REST mode
Terrence Zhang
committed
review: fix remaining issues (stale docstring, unused vars, README defaults)
Terrence Zhang
committed
remove Slurm REST dead code, rewrite README in English
Terrence Zhang
committed
review: normalize Slurm states, implement _logs_cli, dedup image check, add env var docs
Terrence Zhang
committed
fix: remove --slurm-submit-via from integration tests
Terrence Zhang
committed
refactor: move test infra to tests/integration/infra/, delete unused templates
Terrence Zhang
committed
remove untested PD disaggregation code
Terrence Zhang
committed

Commits on Mar 19, 2026

simplify flowsim init: write annotated template instead of argparse
Terrence Zhang
committed
add --config flag to flowsim init
Terrence Zhang
committed
use template files for flowsim init instead of inline strings
Terrence Zhang
committed
update README: reflect template-file init with --config option
Terrence Zhang
committed
docs: streamline READMEs, unify examples, remove legacy manual workflow
Terrence Zhang
committed
format: fix with black
Terrence Zhang
committed
docs: add --existing-ctx and --decode-tokens to all examples, default decode-tokens to 2
Terrence Zhang
committed
refactor: remove PyYAML fallback, make it a core dependency
Terrence Zhang
committed
fix: reject k8s submit when no PVC or hostPath configured
Terrence Zhang
committed
docs: add missing parameters
Terrence Zhang
committed
fix: unique job names, Slurm exclusive GPU, remove list_jobs prefix filter
Terrence Zhang
committed
refactor: restructure CLI into scripts/cli/ subpackage
Terrence Zhang
committed
fix: use runtime:nvidia for slurm compute node GPU access
Terrence Zhang
committed
refactor: rename test_scheduler_local.py → test_scheduler.py, rewrite header
Terrence Zhang
committed
docs: update output structure to include logs, merged, and shape dirs
Terrence Zhang
committed

Commits on Mar 20, 2026

Update scripts/cli/submit.py

TerrenceZhangX
and
Copilot
authored
Update schedulers/local.py

TerrenceZhangX
and
Copilot
authored
Update tests/integration/infra/kind-multi-node.yaml

TerrenceZhangX
and
Copilot
authored
Update tests/integration/infra/slurm-compose.yaml

TerrenceZhangX
and
Copilot
authored
Update tests/integration/infra/slurm.conf

TerrenceZhangX
and
Copilot
authored
fix: parameterize hardcoded host paths in slurm-compose
Terrence Zhang
committed
fix: auto-mount output_dir in docker/enroot container modes
Terrence Zhang
committed