feat(core): PymoniK rewrite#30
Draft
AncientPatata wants to merge 22 commits into
Draft
Conversation
62c2af4 to
2a6c496
Compare
2a6c496 to
c5dc997
Compare
Lazy futures: the completion loop marks status only; result bytes download lazily on .result()/await, once and memoised. Pipelining downloads nothing — reading one terminal of an N-task DAG fetches one result, not N. Result reuse: structural (Merkle) cache keys — a Future arg contributes its upstream's key, so intermediate tasks are cacheable and an unchanged DAG prefix stays stable when a downstream task changes. Source-based fn_identity + cache_version override; client-side key->result_id index validated against the cluster (RESULT_ID + STATUS==COMPLETED). A hit reuses the existing result_id with no resubmission (cross-session verified on real ArmoniK); reused futures are named reused-<task_id>. cache_locally option plumbed; its wiring (Layer 3) is a documented TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pymonik CLI (main/doctor/replay/mcp) lives on ad/rewrite-extras now so this branch can focus on hardening the core runtime. Also drops the now-dangling `pymonik` console-script entry point; `pymonik-worker` stays. Nothing in the library imports pymonik.cli, so removal is clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…multi-result tasks
…ng results to minimize confusion between sync/async
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PymoniK 0.1 proved out the @task decorator model under time pressure but is fragile under the hood: a schema-less magic-string argument protocol, four overlapping ways to set task options, a ~50-line
hand-rolled option merge, double-pickling via LazyArgs, a worker that re-uses the client class via is_worker=True and rebuilds a client per task, polling instead of event streams, and no async,
retries, OTel, local-test story, or typed errors.
This PR is the v1 clean-break rewrite specified: decorator-first ergonomics, typed end-to-end, a single typed wire envelope, first-class local execution, etc.
Description
Greenfield src/pymonik/ package replacing the v1 pymonik/ + pymonik_worker/ + test_client/ trees. Highlights:
retries, and result resolution. Sync + async surfaces (session()/session_async(), .result()/await).
field-driven .merge(); gather/as_completed (+ sync siblings).
(_internal/refs.py) replace 0.1's magic strings; oversize inline args auto-spill to content-addressed blobs.
Testing
multiple unit/local tests (tests/) against LocalCluster and stubbed armonik clients — no network. Strong coverage of TaskOpts.merge, env_id canonicalization, envelope round-trips, MultiResult AST extraction, map/starmap, lazy futures, multi-partition, hooks.
Impact
No direct impact on ArmoniK. Might need to make changes to the existing PymoniK partition.
Additional Information
None yet.
Checklist