Skip to content

feat(core): PymoniK rewrite#30

Draft
AncientPatata wants to merge 22 commits into
mainfrom
ad/rewrite
Draft

feat(core): PymoniK rewrite#30
AncientPatata wants to merge 22 commits into
mainfrom
ad/rewrite

Conversation

@AncientPatata

@AncientPatata AncientPatata commented Apr 26, 2026

Copy link
Copy Markdown
Contributor

Motivation

PymoniK 0.1 proved out the @task decorator model under time pressure but is fragile under the hood: a schema-less magic-string argument protocol, four overlapping ways to set task options, a ~50-line
hand-rolled option merge, double-pickling via LazyArgs, a worker that re-uses the client class via is_worker=True and rebuilds a client per task, polling instead of event streams, and no async,
retries, OTel, local-test story, or typed errors.

This PR is the v1 clean-break rewrite specified: decorator-first ergonomics, typed end-to-end, a single typed wire envelope, first-class local execution, etc.

Description

Greenfield src/pymonik/ package replacing the v1 pymonik/ + pymonik_worker/ + test_client/ trees. Highlights:

  • Client/session split (client.py, session.py): PymonikClient owns the channel/credentials/OTel; Session owns the session id, default options, the multiplexed events stream (with a polling fallback),
    retries, and result resolution. Sync + async surfaces (session()/session_async(), .result()/await).
  • Task model (task.py, options.py, composition.py): @task with ParamSpec/@overload signature preservation; .spawn/.map/.starmap/.with_options/.tail; TaskOpts as a frozen/slots/kw_only dataclass with a
    field-driven .merge(); gather/as_completed (+ sync siblings).
  • Typed wire envelope (envelope.py): one msgspec.Struct with a version field and an embedded client Python-minor guard that raises instead of SIGSEGV'ing on a cloudpickle minor mismatch. Arg refs
    (_internal/refs.py) replace 0.1's magic strings; oversize inline args auto-spill to content-addressed blobs.
  • Futures (future.py, multiresult.py): Future[T]/FutureList[T], lazy materialization (download only on .result()/await), MultiResult/MultiResultHandle/TailPromise for multi-output and tail-call sub-tasking.
  • Worker runtime (worker.py, worker_session.py, _internal/task_runner.py, subprocess_dispatch.py): single boot, shared submission pipeline, in-process splice vs isolate=True subprocess for runtime deps.
  • Cross-cutting: typed PymonikError hierarchy; structlog; optional [otel] trace propagation; public pymonik.hooks; fluent introspection DSL client.tasks.where(...).list(); LocalCluster in-process backend (testing/local.py); content-addressed runtime-dep envs and a structural-key result-reuse cache.
  • Adds .docs/guides/, runnable examples/.py, tests/, worker-image/Dockerfile; removes 0.1 trees and automation.py.

Testing

multiple unit/local tests (tests/) against LocalCluster and stubbed armonik clients — no network. Strong coverage of TaskOpts.merge, env_id canonicalization, envelope round-trips, MultiResult AST extraction, map/starmap, lazy futures, multi-partition, hooks.

Impact

No direct impact on ArmoniK. Might need to make changes to the existing PymoniK partition.

Additional Information

None yet.

Checklist

  • My code adheres to the coding and style guidelines of the project.
  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • [~] I have made corresponding changes to the documentation.
  • I have thoroughly tested my modifications and added tests when necessary.
  • Tests pass locally and in the CI.
  • I have assessed the performance impact of my modifications.

@CLAassistant

CLAassistant commented Apr 26, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@AncientPatata AncientPatata force-pushed the ad/rewrite branch 2 times, most recently from 62c2af4 to 2a6c496 Compare April 26, 2026 14:33
AncientPatata and others added 9 commits May 28, 2026 15:04
Lazy futures: the completion loop marks status only; result bytes
download lazily on .result()/await, once and memoised. Pipelining
downloads nothing — reading one terminal of an N-task DAG fetches one
result, not N.

Result reuse: structural (Merkle) cache keys — a Future arg contributes
its upstream's key, so intermediate tasks are cacheable and an unchanged
DAG prefix stays stable when a downstream task changes. Source-based
fn_identity + cache_version override; client-side key->result_id index
validated against the cluster (RESULT_ID + STATUS==COMPLETED). A hit
reuses the existing result_id with no resubmission (cross-session
verified on real ArmoniK); reused futures are named reused-<task_id>.
cache_locally option plumbed; its wiring (Layer 3) is a documented TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@AncientPatata AncientPatata changed the title feat: PymoniK rewrite (WIP) feat(core): PymoniK rewrite Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants