Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/phase-1-api-schema-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Every HTTP failure from the local API uses this stable envelope. Successful resp

`request_id` is a fresh local trace identifier for correlating a caller response with local process logs. `error.details` is deliberately an empty object in version 1: API failures must not return exception text, stack traces, creator node IDs, manifests, receipts, lineage records, or contribution-attribution data. Local logs retain the original diagnostic.

The current stable codes are `INVALID_INPUT` (400 or 405), `NOT_FOUND` (404 route lookup), `MISSING_MANIFEST` (404), `VALIDATION_FAILURE` (400 or 404 when evidence is absent), `LINEAGE_LOOKUP_FAILURE` (400), `CONTRIBUTION_ATTRIBUTION_FAILURE` (400), and `INTERNAL_ERROR` (500). Callers should test `error.code` rather than response messages or log text.
The current stable codes are `INVALID_INPUT` (400 or 405), `NOT_FOUND` (404 route lookup), `MISSING_MANIFEST` (404), `VALIDATION_FAILURE` (400 or 404 when evidence is absent), `LINEAGE_LOOKUP_FAILURE` (400), `CONTRIBUTION_ATTRIBUTION_FAILURE` (400), `TRACE_LINEAGE_FAILURE` (400), and `INTERNAL_ERROR` (500). Callers should test `error.code` rather than response messages or log text.

## Route index

Expand All @@ -32,6 +32,7 @@ The current stable codes are `INVALID_INPUT` (400 or 405), `NOT_FOUND` (404 rout
| `GET /health`, `/status`, `/api/status`, `/version`, `/node`, `/api/node`, `/peers`, `/api/peers`, `/api/jobs`, `/capabilities`, `/api/capabilities`, `/api/model-manifests`, `/api/package`, `/api/network`, `/logs`, `/api/logs`, `/api/events`, `/`, `/shutdown`, `/restart` | None, except local control signal posts | Current local status/control shape described in the API boundary; none writes provenance. |
| `POST /api/jobs` | Local Job Submission v1 | Local Job Submission Acceptance v1 |
| `GET /api/jobs/{job_id}` | Required path `job_id` | Local Job Status v1 artifact projection |
| `GET /api/jobs/{job_id}/trace` | Required path `job_id` | Local Job Attribution Trace v1 |
| `GET /api/validation-receipts` | None to list receipt metadata; legacy detail lookup accepts exactly one of `receipt_id`, `work_id`, or `latest=true` | Local Validation Receipt List v1 or Local Validation Receipt v5 |
| `GET /api/validation-receipts/{receipt_id}` | Required stable local receipt ID | Local Validation Receipt v5 |
| `GET /api/contributions` | None | Local Contribution Lookup v1 |
Expand Down Expand Up @@ -75,6 +76,8 @@ The only local job states are: `created` (a submission manifest and initial loca

Queued jobs have `null` worker/result/validation evidence. Completed jobs preserve a validation receipt reference and validation-gated attribution; a receipt is local evidence, not consensus. A well-formed but unknown local job ID returns `schema_version`, `job_id`, `status: not_found`, and `error`; malformed path IDs are rejected as invalid input.

`GET /api/jobs/{job_id}/trace` is a read-only local evidence join. Local Job Attribution Trace v1 returns `schema_version: 1`, `trace_scope: local-only-validation-gated-attribution`, the requested `job_id`, and an ordered `chain` containing job, result, validation-receipt, contribution, manifest, and creator-node records. It succeeds only for an accepted, passed local validation receipt whose persisted result hash, manifest reference and hash, creator ID, and contribution attribution agree with the linked local artifacts. Missing or inconsistent links fail closed; the endpoint does not create contribution evidence or imply network consensus.

Example completed projection (dynamic IDs and timestamps omitted):

```json
Expand Down
4 changes: 2 additions & 2 deletions docs/phase-1-local-api-boundary.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Phase 1 supports these operations only:
| Create or load node | `start_local_node_runtime(runtime_dir, reset_creator_identity=False)` | `start-local-node --runtime-dir <dir>` | Create a new local runtime or load its preserved identity and startup artifacts. |
| Inspect node provenance | `inspect_local_node_runtime(runtime_dir)` and `local_node_status(runtime_dir)` | none | Read identity, manifest, receipt, lineage, and attribution references without mutation. |
| Stop or restart node | `stop_local_node_runtime(...)`, `restart_local_node_runtime(...)` | `shutdown-local-node`, `restart-local-node` | Persist local lifecycle evidence while retaining identity and audit links. |
| Submit or inspect one local job | `NodeRuntimeService.submit_local_job(request)`, `get_local_job_status(job_id)`, and `execute_submitted_local_job(job_id, worker_node_id)` | none | Record a local submission, inspect its lifecycle, or run it through the deterministic local helper. HTTP exposes submission at `POST /api/jobs` and inspection at `GET /api/jobs/{job_id}`. |
| Submit or inspect one local job | `NodeRuntimeService.submit_local_job(request)`, `get_local_job_status(job_id)`, `trace_local_job_attribution(job_id)`, and `execute_submitted_local_job(job_id, worker_node_id)` | none | Record a local submission, inspect its lifecycle or validation-gated attribution chain, or run it through the deterministic local helper. HTTP exposes submission at `POST /api/jobs`, status inspection at `GET /api/jobs/{job_id}`, and attribution tracing at `GET /api/jobs/{job_id}/trace`. |
| Submit deterministic local batch work | `dispatch_local_batch_command(manifest_path, message_log_path)` or `run_local_flow(manifest_path, output_dir, ...)` | `dispatch-local-batch` or `run-local-flow` | Validate a version 1 batch manifest, then dispatch or execute it locally. |
| Validate completed local work | `validate_local_node_results(assignment_log_path, result_log_path, validation_log_path)` | `validate-local-results` | Replay local assignment/result logs and write a deterministic validation report. |
| Read validated result history | `audit_local_flow(output_dir)` | `audit-local-flow` | Verify a completed local-flow artifact set. |
Expand Down Expand Up @@ -83,7 +83,7 @@ The runtime owns startup manifests, startup receipts, and startup lineage record

`POST /api/jobs` is a localhost-only submission route backed by `NodeRuntimeService.submit_local_job(request)`. Its version-1 request is a JSON object with required integer `schema_version: 1`, non-empty `job_type`, object `payload`, non-empty `creator_node_id`, non-empty `requested_validation_mode`, required `lineage_parent_refs` (a list of non-empty strings; empty for root lineage), and required object `attribution_metadata` (empty allowed). It writes one immutable `data/job-submissions/<job-id>.json` manifest only after validation and returns `schema_version: 1`, `job_id`, `status: accepted_pending_execution`, `manifest_ref`, `next_validation_expectation: pending_requested_local_validation`, and `network_mode: local-only-no-p2p`. Submission does not dispatch work, create a receipt, complete validation, contact peers, or award credit.

`GET /api/jobs/{job_id}` and `get_local_job_status(job_id)` return the immutable submission provenance plus the current local lifecycle projection. Queued jobs have no invented worker, result, or receipt. The local-only `execute_submitted_local_job(job_id, worker_node_id)` helper persists separate result, validation-receipt, and status records through the existing deterministic runner and validator; completed responses then include the worker, relative result reference and summary, validation receipt/pass-fail metadata, lineage, and validation-gated attribution. Unknown IDs return the stable `{job_id, status: "not_found", error: "local job not found"}` response. This is not a daemon queue or remote dispatch API.
`GET /api/jobs/{job_id}` and `get_local_job_status(job_id)` return the immutable submission provenance plus the current local lifecycle projection. Queued jobs have no invented worker, result, or receipt. `GET /api/jobs/{job_id}/trace` and `trace_local_job_attribution(job_id)` read the completed local job, result, accepted validation receipt, contribution attribution, manifest, and preserved creator identity as one ordered, validation-gated chain; missing or mismatched evidence is rejected rather than inferred. The local-only `execute_submitted_local_job(job_id, worker_node_id)` helper persists separate result, validation-receipt, and status records through the existing deterministic runner and validator; completed responses then include the worker, relative result reference and summary, validation receipt/pass-fail metadata, lineage, and validation-gated attribution. Unknown IDs return the stable `{job_id, status: "not_found", error: "local job not found"}` response. This is not a daemon queue or remote dispatch API.

A local batch submission is an invocation of `dispatch-local-batch` or `run-local-flow` with a version 1 job-batch manifest. It is accepted only after manifest parsing confirms the version, non-empty unique node roster, non-empty jobs, and each job’s non-empty `job_id`/`job_type` plus object payload. Execution additionally rejects unsupported job types.

Expand Down
12 changes: 12 additions & 0 deletions src/aethermesh_core/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@ def _runtime_error_code(
"CONTRIBUTION_ATTRIBUTION_FAILURE",
"Contribution attribution could not be read.",
)
if "trace" in diagnostic:
return (
400,
"TRACE_LINEAGE_FAILURE",
"Local attribution trace evidence is incomplete.",
)
if "lineage" in diagnostic:
return 400, "LINEAGE_LOOKUP_FAILURE", "Lineage evidence could not be read."
if "manifest" in diagnostic:
Expand Down Expand Up @@ -250,6 +256,12 @@ def job_result(job_id: str) -> dict[str, Any]:

return runtime_service.get_local_job_result(job_id)

@app.get("/api/jobs/{job_id}/trace")
def job_attribution_trace(job_id: str) -> dict[str, Any]:
"""Read one validation-gated local attribution chain without mutation."""

return runtime_service.trace_local_job_attribution(job_id)

@app.get("/api/result-reports")
def result_reports() -> dict[str, Any]:
"""List deterministic, metadata-only summaries of local result reports."""
Expand Down
126 changes: 126 additions & 0 deletions src/aethermesh_core/runtime_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -1987,6 +1987,132 @@ def list_local_validation_receipts(self) -> dict[str, Any]:
],
}

def trace_local_job_attribution(self, job_id: str) -> dict[str, Any]:
"""Return one validation-gated local job attribution chain.

This is an inspection-only join over persisted evidence. Every link is
checked directly; attribution is never inferred from loose metadata.
"""
if not self._is_local_job_id(job_id):
raise RuntimeServiceError("job_id must be a local job ID")
manifest_ref = f"data/job-submissions/{job_id}.json"
manifest_path = self.paths.home / manifest_ref
if not manifest_path.exists():
raise RuntimeServiceError("trace missing required local job manifest")
manifest = self._load_local_job_document(
manifest_path, "job submission manifest"
)
manifest_job = manifest.get("job")
creator_node_id = manifest.get("creator_node_id")
if (
not isinstance(manifest_job, dict)
or manifest_job.get("job_id") != job_id
or not isinstance(creator_node_id, str)
or not creator_node_id
):
raise RuntimeServiceError("trace has invalid local job manifest linkage")

result = self.get_local_job_result(job_id)
try:
receipt = self.get_local_validation_receipt(work_id=job_id)
except ValidationReceiptNotFoundError:
raise
except RuntimeServiceError as exc:
raise RuntimeServiceError(
"trace validation receipt is missing or has invalid lineage evidence"
) from exc
status_path = self._job_status_path(job_id)
if not status_path.exists():
raise RuntimeServiceError("trace missing required local job status")
status = self._load_local_job_document(status_path, "job status record")
contribution = status.get("contribution_attribution")
expected_receipt_id = self._receipt_id_for_job(job_id)
expected_result_id = f"local-result-{job_id}"
manifest_id = canonical_json_hash(manifest, prefix="sha256:")
contribution_id = f"local-contribution-{job_id}"

if receipt["status"] != "accepted" or receipt["validation_status"] != "passed":
raise RuntimeServiceError(
"trace requires an accepted passed validation receipt"
)
if (
result.get("result_id") != expected_result_id
or result.get("job_id") != job_id
or result.get("validation_receipt_id") != expected_receipt_id
or result.get("references", {}).get("validation_receipt_ids")
!= [expected_receipt_id]
or result.get("manifest_id") != manifest_id
or result.get("references", {}).get("manifest_hash") != manifest_id
or result.get("creator_node_id") != creator_node_id
):
raise RuntimeServiceError(
"trace result does not match job manifest lineage"
)
if (
receipt["receipt_id"] != expected_receipt_id
or receipt["job_id"] != job_id
or receipt["manifest_ref"] != manifest_ref
or receipt["result_hash"] != result.get("result_hash")
or receipt["creator_node_id"] != creator_node_id
):
raise RuntimeServiceError(
"trace validation receipt does not match result lineage"
)
if (
not isinstance(contribution, dict)
or contribution.get("job_id") != job_id
or contribution.get("creator_node_id") != creator_node_id
or contribution != receipt["contribution_attribution"]
):
raise RuntimeServiceError(
"trace contribution does not match validation receipt"
)

return {
"schema_version": 1,
"trace_scope": "local-only-validation-gated-attribution",
"job_id": job_id,
"chain": [
{
"record_type": "job",
"job_id": job_id,
"result_id": expected_result_id,
"manifest_ref": manifest_ref,
},
{
"record_type": "result",
"result_id": expected_result_id,
"job_id": result["job_id"],
"validation_receipt_id": result["validation_receipt_id"],
"manifest_id": result["manifest_id"],
},
{
"record_type": "validation_receipt",
"validation_receipt_id": receipt["receipt_id"],
"job_id": receipt["job_id"],
"result_id": expected_result_id,
"manifest_ref": receipt["manifest_ref"],
"contribution_id": contribution_id,
},
{
"record_type": "contribution",
"contribution_id": contribution_id,
"job_id": contribution["job_id"],
"validation_receipt_id": expected_receipt_id,
"manifest_ref": manifest_ref,
"creator_node_id": contribution["creator_node_id"],
},
{
"record_type": "manifest",
"manifest_id": manifest_id,
"manifest_ref": manifest_ref,
"job_id": manifest_job["job_id"],
"creator_node_id": creator_node_id,
},
{"record_type": "creator_node", "creator_node_id": creator_node_id},
],
}

@staticmethod
def _receipt_id_for_job(job_id: str) -> str:
return f"{VALIDATION_RECEIPT_ID_PREFIX}{job_id}"
Expand Down
82 changes: 82 additions & 0 deletions tests/test_runtime_service_api_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2535,6 +2535,88 @@ def test_local_job_result_read_path_returns_stored_report_and_preserves_identity
)
self.assertEqual(report_path.read_bytes(), report_before_read)

def test_local_job_attribution_trace_is_validation_gated_and_fails_closed(
self,
) -> None:
request, _expected_output = _valid_local_work_fixture()
request["creator_node_id"] = "creator-byte-preserved"
with tempfile.TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
service = NodeRuntimeService.from_home(root)
submission = service.submit_local_job(request)
service.execute_submitted_local_job(
submission["job_id"], "worker-local-fixture"
)
evidence_before = {
path: path.read_bytes() for path in (root / "data").rglob("*.json")
}

trace = service.trace_local_job_attribution(submission["job_id"])

self.assertEqual(
trace["trace_scope"], "local-only-validation-gated-attribution"
)
self.assertEqual(
[item["record_type"] for item in trace["chain"]],
[
"job",
"result",
"validation_receipt",
"contribution",
"manifest",
"creator_node",
],
)
job, result, receipt, contribution, manifest, creator = trace["chain"]
self.assertEqual(job["result_id"], result["result_id"])
self.assertEqual(
result["validation_receipt_id"], receipt["validation_receipt_id"]
)
self.assertEqual(
receipt["contribution_id"], contribution["contribution_id"]
)
self.assertEqual(contribution["manifest_ref"], manifest["manifest_ref"])
self.assertEqual(creator["creator_node_id"], request["creator_node_id"])
self.assertEqual(
evidence_before,
{path: path.read_bytes() for path in (root / "data").rglob("*.json")},
)

api = create_app(service)

async def fetch_trace() -> httpx.Response:
transport = httpx.ASGITransport(app=api)
async with httpx.AsyncClient(
transport=transport, base_url="http://testserver"
) as client:
return await client.get(f"/api/jobs/{submission['job_id']}/trace")

response = asyncio.run(fetch_trace())
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), trace)

receipt_path = (
root
/ "data"
/ "job-validation-receipts"
/ f"{submission['job_id']}.json"
)
stored_receipt = json.loads(receipt_path.read_text(encoding="utf-8"))
stored_receipt["manifest_ref"] = "data/job-submissions/other.json"
receipt_path.write_text(json.dumps(stored_receipt), encoding="utf-8")
with self.assertRaisesRegex(
RuntimeServiceError, "trace validation receipt"
):
service.trace_local_job_attribution(submission["job_id"])
failed_response = asyncio.run(fetch_trace())
self.assertEqual(failed_response.status_code, 400)
self.assertEqual(
failed_response.json()["error"]["code"], "TRACE_LINEAGE_FAILURE"
)
receipt_path.unlink()
with self.assertRaisesRegex(RuntimeServiceError, "receipt not found"):
service.trace_local_job_attribution(submission["job_id"])

def test_execution_writes_a_provenanced_result_report_only_after_runner_finishes(
self,
) -> None:
Expand Down
Loading