Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,26 @@ jobs:
|| { echo "FAIL: streaming demo did not pass its assertions"; exit 1; }
echo "v0.2 Streaming exit criterion holds: $CHUNKS chunks live, one recorded completion, replay with zero model calls."

# The v0.2 timeline exit criterion (spec §12): an execution's folded state renders to a read-only
# HTML report. "self-contained: true" is the load-bearing assertion — a report with no scripts and
# no external references is one you can attach to a ticket or open from file:// years later.
# "deterministic: true" pins that it stays a pure fold of the log, not a live view.
- name: v0.2 Timeline exit demo (folded execution renders to a self-contained HTML report)
run: |
set -uo pipefail
CP="catalyst-api/target/classes:$(cat catalyst-api/target/cp.txt)"

echo "--- render an execution's timeline ---"
OUT="$(java -cp "$CP" com.cajunsystems.catalyst.api.Demo timeline)"
echo "$OUT"
echo "$OUT" | grep -q "self-contained (no scripts, no external refs): true" \
|| { echo "FAIL: the report referenced something external"; exit 1; }
echo "$OUT" | grep -q "deterministic for a given log: true" \
|| { echo "FAIL: the report is not a pure fold of the log"; exit 1; }
echo "$OUT" | grep -q "\[timeline\] timeline criterion holds" \
|| { echo "FAIL: timeline demo did not pass its assertions"; exit 1; }
echo "v0.2 Timeline exit criterion holds: folded execution rendered to a portable HTML report."

# The -javaagent launch the README documents, exercised the way a user would: the agent jar alone,
# with NO Byte Buddy on the application classpath (a -javaagent jar does not get its Maven
# dependencies). This gates two failure modes that are invisible to the in-process tests — an
Expand Down
10 changes: 10 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ intended source, but if `jitpack.io` is blocked, install Gumbo locally first:
task packages into calls on `catalyst-core`'s `AutoCapture` bridge. The bridge lives in **core**, not
here, so instrumented classes link without the agent. Depends on Byte Buddy; tests fork a JVM per
class (agent installs are JVM-wide).
- `catalyst-timeline` — `TimelineReport`: renders a folded `ExecutionState` as a self-contained HTML
page (header + roll-ups + step table). Read-only, post-hoc, pure function of the log; depends on
`catalyst-core` only — no templating engine, no web server. All interpolated log content is escaped.
- `catalyst-otel` — `CatalystTracer`: folds an execution's event log into an OpenTelemetry trace
(root span + per-boundary child spans + lifecycle annotations). Read-only, post-hoc, no runtime hook.
Depends on the OpenTelemetry **API** only; the app supplies the SDK + exporter. Tested offline with
Expand Down Expand Up @@ -134,6 +137,13 @@ intended source, but if `jitpack.io` is blocked, install Gumbo locally first:
executes between `CompletionRequested` and `CompletionReceived`, and a capture there would append
into that gap and break replay. `LangChain4jModel.streaming(...)` bridges LangChain4j's async
callbacks over a `BlockingQueue` so the sink runs on the **task's** thread, not the provider's.
- **v0.2 Timeline UI** — `TimelineAcceptanceTest` + `Demo timeline`: a folded execution renders to a
self-contained HTML report via `TimelineReport.html(state)` / `writeTo(state, path)`. Same shape as
the OTel exporter: consumes `inspect(id)` (a fold), no runtime hook, so the log stays the only source
of truth. Two properties are gated because they are what make a report useful: **self-contained**
(no scripts/external refs — portable, openable from `file://`) and **deterministic** (a pure fold, so
reports diff cleanly). Everything interpolated is HTML-escaped — tool names, effect labels and
payloads are log content. Oversized payloads are elided at 2000 chars rather than inlined.
- **v0.2 Observability / OTel exporter** — `OtelAcceptanceTest` + `Demo otel`: an execution's event log
folds into one OpenTelemetry trace via `catalyst-otel`'s `CatalystTracer.export(id, events)` — a root
span for the run, a child span per boundary (model/tool/effect/memory; model/tool carry real latency,
Expand Down
38 changes: 35 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ log) provides durability.
incrementally while recording the assembled result, so a streamed execution writes the same log a
non-streaming one does and replays with zero provider calls. See
[Streaming](#streaming-completions).
- **Timeline report (v0.2)** — `TimelineReport.html(runtime.inspect(id))` renders an execution as a
self-contained HTML page: status, token/cost roll-ups, and the step-by-step trajectory. Read-only and
post-hoc, so any recorded execution renders. See [Timeline reports](#timeline-reports).
- **Auto-capture (v0.2)** — attach `catalyst-agent` and task code no longer has to wrap its
nondeterminism: `Instant.now()`, `UUID.randomUUID()`, `Math.random()` and `Random` draws are
rewritten at their call sites into recorded boundaries, so a task written with plain JDK calls
Expand All @@ -79,6 +82,7 @@ Deferred to later milestones (schema slots already reserved so no breaking chang
| `catalyst-tools` | `ClockTool`, `CalculatorTool` |
| `catalyst-langchain4j` | `LangChain4jModel`: adapts any LangChain4j `ChatModel` to Catalyst's `Model` |
| `catalyst-otel` | `CatalystTracer`: folds an execution's log into an OpenTelemetry trace (API-only; app supplies the SDK) |
| `catalyst-timeline` | `TimelineReport`: renders a folded execution as a self-contained HTML timeline (no deps beyond core) |
| `catalyst-agent` | `AutoCaptureAgent`: rewrites the JDK's nondeterministic call sites in task code into recorded boundaries |
| `catalyst-api` | Thin facade: `Catalyst.embedded(path)`, builders, `Serializers` |

Expand Down Expand Up @@ -236,6 +240,34 @@ Two things to know:
request and its result, which no replay can match.

Asserted automatically in `StreamingAcceptanceTest` and gated in CI by `Demo streaming`.
## Timeline reports

The log already contains everything an execution did, so a timeline is a fold, not an instrumentation
layer. `catalyst-timeline` renders that fold as a page:

```java
ExecutionState state = runtime.inspect(id);
TimelineReport.writeTo(state, Path.of("build/reports/" + id.value() + ".html"));
```

You get a status header, the roll-ups from `timelineView()` (model/tool calls, prompt and completion
tokens, cost, boundary latency, wall clock), and the step-by-step trajectory — each boundary with its
kind, label, offset from start, latency, and recorded payload in a collapsed block.

The design constraints are the interesting part:

- **Read-only and post-hoc**, exactly like the OTel exporter. It consumes an `ExecutionState` and
installs no runtime hook, so an execution recorded months ago renders just as well as a fresh one.
- **A pure function of the log.** Two renders of the same execution are byte-identical, which is what
makes a report safe to diff or commit as a build artifact.
- **Self-contained.** Inline CSS, no scripts, no fonts, no images — it opens from a `file://` URL with
nothing else present, so you can attach one to a ticket.
- **Everything is escaped.** Tool names, effect labels and recorded payloads are log content, and a
report is likely to be opened by someone other than whoever produced the execution.

The module depends on `catalyst-core` and nothing else: no templating engine, no web server.

Asserted automatically in `TimelineAcceptanceTest` and gated in CI by `Demo timeline`.

## Auto-capture: nondeterminism without the ceremony

Expand Down Expand Up @@ -302,9 +334,9 @@ and **M2** (branch + diff).

## Roadmap

v0.1 is complete. See [ROADMAP.md](ROADMAP.md) for what's next — v0.2 (largely shipped: snapshots,
blob store, retry semantics, the OTel exporter, built-in tools, the auto-capture agent, streaming
completions; remaining: the timeline UI) and v1 (agents, the
v0.1 is complete, and v0.2 with it. See [ROADMAP.md](ROADMAP.md) for what's next — v0.2 (shipped:
snapshots, blob store, retry semantics, the OTel exporter, built-in tools, the auto-capture agent,
streaming completions, the timeline report) and v1 (agents, the
`WAITING`/signal APIs and human-in-the-loop via Boudin, distributed execution over a Gumbo cluster,
and the replay-driven eval harness).

Expand Down
20 changes: 17 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ auto-capture agent, per-execution locking, streaming, and observability sequence
guide, not a contract — it flexes as we learn. Snapshots, the cancellation event (①), the task
registry / standalone `resume(id)` (②), the built-in HTTP + Filesystem tools (③), generic-collection
payloads (④), the blob store (⑤), schema evolution, per-execution locking, retry semantics, the OTel
exporter, the auto-capture agent, and streaming completions have shipped; the timeline UI is the last
remaining v0.2 item.
exporter, the auto-capture agent, streaming completions, and the timeline UI have all shipped —
**v0.2 is complete**.

### Durability & storage (spec §8)
- ✅ **Snapshots** — periodic fold checkpoints so long executions don't re-fold the whole log on
Expand Down Expand Up @@ -186,7 +186,21 @@ remaining v0.2 item.
`runtime.log().read(id)`, needing no runtime hook — so the log genuinely *is* the trace. The module
depends on the OpenTelemetry **API** only; the app supplies the SDK + a real OTLP exporter (the same
shape as the LangChain4j adapter). Gated by the v0.2 OTel exit demo in CI.
- **Timeline UI** — a read-only view over `inspect(id).timelineView()` / `Trajectory`.
- ✅ **Timeline UI** — `catalyst-timeline`'s `TimelineReport.html(state)` renders a folded
`ExecutionState` as a **self-contained HTML page**: a status header, the roll-ups
(`timelineView()` — model/tool counts, tokens, cost, latency, wall clock) and the step-by-step
trajectory with each boundary's label, offset, latency and recorded payload. Read-only and post-hoc,
the same shape as the OTel exporter — it consumes a fold of the log and installs no runtime hook, so
any execution renders without having cooperated in advance, and the log stays the single source of
truth. Because the input is a fold, the output is a **pure function** of the log: two renders of one
execution are byte-identical, which is what makes a report safe to diff or commit as a build
artifact. The page references nothing external (inline CSS, no scripts/fonts/images) so it opens from
`file://` years later; everything interpolated is HTML-escaped, since tool names, effect labels and
recorded payloads are log content and a report gets opened in a browser by someone who did not
produce the execution. The module depends on `catalyst-core` alone — no templating engine, no web
server. Gated by the v0.2 Timeline exit demo in CI. *Deferred (noted): latency bars and a
`TrajectoryDiff` view (the M2 branch comparison) — the table is the read-only view the roadmap asked
for; visual timing and diff rendering are polish on top.*

---

Expand Down
4 changes: 4 additions & 0 deletions catalyst-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@
<groupId>com.cajunsystems</groupId>
<artifactId>catalyst-otel</artifactId>
</dependency>
<dependency>
<groupId>com.cajunsystems</groupId>
<artifactId>catalyst-timeline</artifactId>
</dependency>
<!--
The auto-capture exit demo attaches the agent in-process. Optional for the same reason as the
OTel SDK below: an app that never wants bytecode instrumentation should not inherit Byte Buddy
Expand Down
72 changes: 72 additions & 0 deletions catalyst-api/src/main/java/com/cajunsystems/catalyst/api/Demo.java
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import com.cajunsystems.catalyst.mock.MockModel;
import com.cajunsystems.catalyst.otel.CatalystTracer;
import com.cajunsystems.catalyst.runtime.CatalystRuntime;
import com.cajunsystems.catalyst.timeline.TimelineReport;
import com.cajunsystems.catalyst.runtime.ExecutionHandle;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
Expand Down Expand Up @@ -119,6 +120,8 @@ public static void main(String[] args) throws Exception {
autoCaptureDemo(Files.createTempDirectory("catalyst-autocapture-"));
} else if (args.length >= 1 && args[0].equals("streaming")) {
streamingDemo(Files.createTempDirectory("catalyst-streaming-"));
} else if (args.length >= 1 && args[0].equals("timeline")) {
timelineDemo(Files.createTempDirectory("catalyst-timeline-"));
} else if (args.length >= 1 && args[0].equals("schema")) {
schemaDemo();
} else {
Expand Down Expand Up @@ -733,6 +736,75 @@ private static void streamingDemo(Path dir) {
}
}

/**
* The v0.2 timeline exit demo (spec §12): an execution's folded state is rendered as a
* self-contained HTML report — the read-only view over {@code inspect(id)}.
*
* <p>Like the OTel exporter, this is post-hoc and read-only: it consumes an {@code ExecutionState},
* which is itself only a fold of the log, and installs no runtime hook. So the report needs no
* cooperation from the execution that produced it — any log, live or long finished, renders.
*/
private static void timelineDemo(Path dir) throws Exception {
MockModel model = summarizerModel();
Path sandbox = Files.createDirectory(dir.resolve("sandbox"));

com.cajunsystems.catalyst.tools.FilesystemTool fs =
new com.cajunsystems.catalyst.tools.FilesystemTool(sandbox);

// A task with one of every boundary kind, so the report has something to show.
Task<String> mixed = ctx -> {
String summary = ctx.model().complete(STEP1).message();
ctx.call(fs, com.cajunsystems.catalyst.tools.FilesystemTool.Command.write("out.txt", summary));
ctx.memory().put("summary", summary);
String stamp = ctx.effect("stamp", () -> "2024-01-01T00:00:00Z");
String finalAnswer = ctx.model().complete(STEP2).message();
return summary + "|" + finalAnswer + "|" + stamp;
};

try (CatalystRuntime runtime = Catalyst.builder()
.log(GumboEventLog.at(Files.createDirectory(dir.resolve("log"))))
.model(model)
.costModel(CostModel.perMillionTokens(3.0, 15.0))
.build()) {

ExecutionHandle<String> handle = runtime.execute(mixed, ExecutionOptions.withKey(KEY));
handle.result();
ExecutionState state = runtime.inspect(handle.id());

Path report = TimelineReport.writeTo(state, dir.resolve("timeline.html"));
String html = Files.readString(report);
Timeline t = state.timelineView();

System.out.println("[timeline] recorded execution " + handle.id().value());
System.out.println("[timeline] steps folded: " + state.timeline().size()
+ " (" + t.modelCalls() + " model, " + t.toolCalls() + " tool)");
System.out.println("[timeline] report written: " + report + " (" + html.length() + " bytes)");

// Self-contained is the property that makes a report portable — attach it to a ticket,
// commit it as a build artifact, open it from file:// with nothing else present.
boolean selfContained = !html.contains("http://") && !html.contains("https://")
&& !html.contains("<script") && !html.contains("<link") && !html.contains("<img");
System.out.println("[timeline] self-contained (no scripts, no external refs): " + selfContained);

// The report is a pure function of the fold, so the same log always renders the same page.
boolean deterministic = TimelineReport.html(state).equals(TimelineReport.html(state));
System.out.println("[timeline] deterministic for a given log: " + deterministic);

if (!selfContained) throw new AssertionError("report referenced something external");
if (!deterministic) throw new AssertionError("report is not a pure fold of the log");
for (String kind : new String[]{"MODEL", "TOOL", "EFFECT", "MEMORY_WRITE", "COMPLETED"}) {
if (!html.contains(">" + kind + "<")) {
throw new AssertionError("report is missing a " + kind + " step");
}
}
if (!html.contains(handle.id().value())) {
throw new AssertionError("report does not identify the execution");
}
System.out.println("[timeline] timeline criterion holds: the folded execution rendered to a"
+ " self-contained, deterministic HTML report.");
}
}

/**
* The v0.2 auto-capture exit demo (spec §6): the <em>same task class</em>, containing plain
* {@code Instant.now()} / {@code UUID.randomUUID()} / {@code Random} calls and not one
Expand Down
Loading
Loading