-
Notifications
You must be signed in to change notification settings - Fork 1
feat: Add CLIReporterPlugin for real-time progress reporting #252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| ## ADDED Requirements | ||
|
|
||
| ### Requirement: Interactive CLI Progress Reporting | ||
|
|
||
| The `CLIReporterPlugin` SHALL provide real-time updates of task states during workflow execution. | ||
|
|
||
| #### Scenario: Displaying running tasks | ||
|
|
||
| - **WHEN** a task starts execution and triggers `taskStart` | ||
| - **THEN** the CLI reporter SHALL display the task name with a "running" indicator. | ||
|
|
||
| #### Scenario: Displaying successful tasks | ||
|
|
||
| - **WHEN** a task finishes successfully and triggers `taskEnd` with a `success` status | ||
| - **THEN** the CLI reporter SHALL display the task name with a "success" indicator and execution duration. | ||
|
|
||
| #### Scenario: Displaying failed tasks | ||
|
|
||
| - **WHEN** a task finishes with an error and triggers `taskEnd` with a `failure` status | ||
| - **THEN** the CLI reporter SHALL display the task name with a "failure" indicator and error details. | ||
|
|
||
| #### Scenario: Displaying skipped tasks | ||
|
|
||
| - **WHEN** a task is skipped due to unmet dependencies or conditions | ||
| - **THEN** the CLI reporter SHALL display the task name with a "skipped" indicator. | ||
|
|
||
| ### Requirement: Workflow Execution Summary | ||
|
|
||
| The `CLIReporterPlugin` SHALL display a summary table or block upon completion of the entire task graph execution. | ||
|
|
||
| #### Scenario: Generating final summary | ||
|
|
||
| - **WHEN** all tasks have resolved (either success, failure, or skipped) | ||
| - **THEN** the CLI reporter SHALL output the total execution time, along with counts for successful, failed, and skipped tasks. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| import { Plugin, PluginContext } from "../contracts/Plugin.js"; | ||
|
|
||
| /** | ||
| * A plugin that provides real-time CLI observability of workflow progress. | ||
| * It hooks into lifecycle events to display task states and a final summary. | ||
| */ | ||
| export class CLIReporterPlugin<TContext> implements Plugin<TContext> { | ||
| public readonly name = "cli-reporter"; | ||
| public readonly version = "1.0.0"; | ||
|
|
||
| private startTime = 0; | ||
| private successful = 0; | ||
| private failed = 0; | ||
| private skipped = 0; | ||
|
|
||
| public install(context: PluginContext<TContext>): void { | ||
| context.events.on("workflowStart", () => { | ||
| this.startTime = performance.now(); | ||
| this.successful = 0; | ||
| this.failed = 0; | ||
| this.skipped = 0; | ||
| }); | ||
|
|
||
| context.events.on("taskStart", ({ step }) => { | ||
| console.log(`[RUNNING] Task ${step.name}`); | ||
| }); | ||
|
|
||
| context.events.on("taskEnd", ({ step, result }) => { | ||
| if (result.status === "success") { | ||
| this.successful++; | ||
| const durationStr = result.metrics?.duration !== undefined ? ` (${Math.round(result.metrics.duration)}ms)` : ""; | ||
|
Check warning on line 31 in src/plugins/CLIReporterPlugin.ts
|
||
| console.log(`[SUCCESS] Task ${step.name}${durationStr}`); | ||
| } else if (result.status === "failure") { | ||
| this.failed++; | ||
| const errorMsg = result.error ? ` - ${result.error}` : ""; | ||
| console.log(`[FAILURE] Task ${step.name}${errorMsg}`); | ||
| } else if (result.status === "cancelled") { | ||
| // According to the spec we track skipped, but cancelled is another status. | ||
| // Will just increment failed for now or leave it. We'll track it as skipped. | ||
| this.skipped++; | ||
| console.log(`[CANCELLED] Task ${step.name}`); | ||
| } else { | ||
| this.skipped++; | ||
| console.log(`[SKIPPED] Task ${step.name}`); | ||
| } | ||
|
Comment on lines
+37
to
+45
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The logic in taskEnd for handling cancelled and other statuses overlaps with the taskSkipped listener. Specifically, both the else block here and the taskSkipped listener (lines 48-51) increment the skipped counter and log a [SKIPPED] message. If the runner emits both events for a single task (e.g., a task that is skipped but still triggers a terminal event), it will result in duplicate logs and double-counting in the summary. Additionally, the else block acts as a catch-all that logs [SKIPPED] for any unknown status, which might mask future status types (like timeout). |
||
| }); | ||
|
|
||
| context.events.on("taskSkipped", ({ step }) => { | ||
| this.skipped++; | ||
| console.log(`[SKIPPED] Task ${step.name}`); | ||
| }); | ||
|
|
||
| context.events.on("workflowEnd", () => { | ||
| const endTime = performance.now(); | ||
| const totalTime = endTime - this.startTime; | ||
|
|
||
| console.log("--- Workflow Execution Summary ---"); | ||
| console.log(`Total Time: ${Math.round(totalTime)}ms`); | ||
| console.log(`Successful: ${this.successful}`); | ||
| console.log(`Failed: ${this.failed}`); | ||
| console.log(`Skipped: ${this.skipped}`); | ||
| console.log("----------------------------------"); | ||
| }); | ||
|
Comment on lines
+53
to
+63
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Instead of relying on manually incremented counters which are prone to concurrency issues, leverage the results Map passed in the workflowEnd event payload to generate the summary. This ensures the counts are derived from the source of truth for the workflow's completion state. While this involves creating an array from the map values, the performance impact is negligible for a reporter compared to the gain in reliability. context.events.on("workflowEnd", ({ results }) => {
const endTime = performance.now();
const totalTime = endTime - this.startTime;
const counts = Array.from(results.values()).reduce(
(acc, res) => {
if (res.status === "success") acc.successful++;
else if (res.status === "failure") acc.failed++;
else acc.skipped++;
return acc;
},
{ successful: 0, failed: 0, skipped: 0 }
);
console.log("--- Workflow Execution Summary ---");
console.log("Total Time: " + Math.round(totalTime) + "ms");
console.log("Successful: " + counts.successful);
console.log("Failed: " + counts.failed);
console.log("Skipped: " + counts.skipped);
console.log("----------------------------------");
});References
|
||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,185 @@ | ||
| import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; | ||
| import { CLIReporterPlugin } from "../../src/plugins/CLIReporterPlugin.js"; | ||
| import { EventBus } from "../../src/EventBus.js"; | ||
| import { PluginContext } from "../../src/contracts/Plugin.js"; | ||
|
|
||
| describe("CLIReporterPlugin", () => { | ||
| let logSpy: ReturnType<typeof vi.spyOn>; | ||
| let events: EventBus<unknown>; | ||
| let context: PluginContext<unknown>; | ||
|
|
||
| beforeEach(() => { | ||
| logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); | ||
| events = new EventBus<unknown>(); | ||
| context = { events }; | ||
|
|
||
| // Explicitly mock performance.now to return predictable times | ||
| let time = 1000; | ||
| vi.spyOn(performance, "now").mockImplementation(() => { | ||
| const current = time; | ||
| time += 500; | ||
| return current; | ||
| }); | ||
| }); | ||
|
|
||
| afterEach(() => { | ||
| vi.restoreAllMocks(); | ||
| }); | ||
|
|
||
| it("should have correct name and version", () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| expect(plugin.name).toBe("cli-reporter"); | ||
| expect(plugin.version).toBe("1.0.0"); | ||
| }); | ||
|
|
||
| it("should log taskStart correctly", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| // Instead of simulating through a runner, we just emit directly on EventBus | ||
| events.emit("taskStart", { | ||
| step: { name: "Step1", run: async () => ({ status: "success" }) } | ||
| }); | ||
|
|
||
| // Wait a tick for microtask to execute | ||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[RUNNING] Task Step1"); | ||
| }); | ||
|
|
||
| it("should log taskSkipped correctly", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskSkipped", { | ||
| step: { name: "Step1", run: async () => ({ status: "success" }) }, | ||
| result: { status: "skipped" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[SKIPPED] Task Step1"); | ||
| }); | ||
|
|
||
| it("should log taskEnd success correctly", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "success" }) }, | ||
| result: { status: "success", metrics: { startTime: 0, endTime: 100, duration: 100 } } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[SUCCESS] Task Step1 (100ms)"); | ||
| }); | ||
|
|
||
| it("should log taskEnd success correctly without metrics", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "success" }) }, | ||
| result: { status: "success" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[SUCCESS] Task Step1"); | ||
| }); | ||
|
|
||
| it("should log taskEnd failure correctly", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "failure" }) }, | ||
| result: { status: "failure", error: "Some error" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[FAILURE] Task Step1 - Some error"); | ||
| }); | ||
|
|
||
| it("should log taskEnd failure correctly without error message", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "failure" }) }, | ||
| result: { status: "failure" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[FAILURE] Task Step1"); | ||
| }); | ||
|
|
||
| it("should log taskEnd cancelled correctly", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "cancelled" }) }, | ||
| result: { status: "cancelled" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[CANCELLED] Task Step1"); | ||
| }); | ||
|
|
||
| it("should log taskEnd unknown status correctly as skipped", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| // Bypass TS type checking to simulate a runtime unknown status | ||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "skipped" }) }, | ||
| result: { status: "unknown" } as unknown as { status: "success" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("[SKIPPED] Task Step1"); | ||
| }); | ||
|
|
||
| it("should reset counters on workflowStart and log summary on workflowEnd", async () => { | ||
| const plugin = new CLIReporterPlugin(); | ||
| plugin.install(context); | ||
|
|
||
| events.emit("workflowStart", { context: {}, steps: [] }); | ||
| await Promise.resolve(); | ||
|
|
||
| events.emit("taskEnd", { | ||
| step: { name: "Step1", run: async () => ({ status: "success" }) }, | ||
| result: { status: "success" } | ||
| }); | ||
| events.emit("taskEnd", { | ||
| step: { name: "Step2", run: async () => ({ status: "failure" }) }, | ||
| result: { status: "failure" } | ||
| }); | ||
| events.emit("taskSkipped", { | ||
| step: { name: "Step3", run: async () => ({ status: "success" }) }, | ||
| result: { status: "skipped" } | ||
| }); | ||
| events.emit("taskEnd", { | ||
| step: { name: "Step4", run: async () => ({ status: "cancelled" }) }, | ||
| result: { status: "cancelled" } | ||
| }); | ||
|
|
||
| await Promise.resolve(); | ||
|
|
||
| events.emit("workflowEnd", { context: {}, results: new Map() }); | ||
| await Promise.resolve(); | ||
|
|
||
| expect(logSpy).toHaveBeenCalledWith("--- Workflow Execution Summary ---"); | ||
| expect(logSpy).toHaveBeenCalledWith("Total Time: 500ms"); // 1st call to performance.now is in workflowStart (1000), 2nd is in workflowEnd (1500) | ||
| expect(logSpy).toHaveBeenCalledWith("Successful: 1"); | ||
| expect(logSpy).toHaveBeenCalledWith("Failed: 1"); | ||
| expect(logSpy).toHaveBeenCalledWith("Skipped: 2"); // 1 skipped + 1 cancelled | ||
| expect(logSpy).toHaveBeenCalledWith("----------------------------------"); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The plugin maintains execution state (counters and start time) as instance properties. This makes the plugin unsafe for concurrent workflow executions on the same runner instance (or if the plugin instance is shared across multiple runners). If two workflows run concurrently, their counters will be interleaved and the startTime will be overwritten, leading to corrupted summaries.
To improve robustness, consider calculating the final summary statistics directly from the results Map provided in the workflowEnd event payload, which ensures the summary reflects the actual final state of that specific execution.
References