feat: add multi-workflow application config and cross-workflow invocation#133
feat: add multi-workflow application config and cross-workflow invocation#133
Conversation
…tion (#80) - Add ApplicationConfig format for multi-workflow applications referencing separate workflow YAML files with shared module registry - Add step.workflow_call pipeline step for cross-workflow invocation with sync (call-and-wait) and async (fire-and-forget) modes - Add input/output mapping with template expansion for data passing - Add pipelineRegistry to StdEngine for runtime pipeline lookup - Add BuildFromApplicationConfig() with module name conflict detection - Add auto-detection of application configs in cmd/server - Add step.workflow_call schema to module schemas - Add example multi-workflow app (chat platform with 3 workflows) - Add comprehensive tests for workflow call step and multi-config loading Closes #80 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This pull request adds support for multi-workflow applications where multiple workflow YAML files can be composed into a single application with cross-workflow invocation capabilities. The implementation introduces:
Changes:
- ApplicationConfig format for referencing multiple workflow files with auto-detection in cmd/server startup
- step.workflow_call pipeline step supporting sync/async modes with input/output template mapping
- pipelineRegistry in StdEngine for runtime pipeline lookup enabling cross-workflow calls
- MergeApplicationConfig and BuildFromApplicationConfig methods with module name conflict detection
- Comprehensive test coverage (482 lines of tests) and example multi-workflow chat platform demonstrating the feature
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| schema/schema.go | Adds step.workflow_call to core module types list |
| schema/module_schema.go | Defines complete schema for step.workflow_call including config fields, defaults, and documentation |
| plugins/pipelinesteps/plugin.go | Registers step.workflow_call in plugin manifest |
| plugins/api/plugin.go | Reformats newWorkflowRegistry constructor for consistency (no functional change) |
| module/pipeline_step_workflow_call.go | Implements WorkflowCallStep with sync/async modes, template mapping, and timeout support |
| module/pipeline_step_workflow_call_test.go | Comprehensive test coverage for all workflow call scenarios including edge cases |
| config/config.go | Adds ApplicationConfig, WorkflowRef, LoadApplicationConfig, IsApplicationConfig, and MergeApplicationConfig |
| engine.go | Adds pipelineRegistry map, registers step.workflow_call factory closure, implements BuildFromApplicationConfig |
| engine_multi_config_test.go | Extensive integration tests for multi-workflow features and cross-pipeline invocation |
| cmd/server/main.go | Adds config auto-detection, setupFromAppConfig, and loadConfig returns tuple |
| cmd/server/main_test.go | Updates tests to handle new loadConfig return signature |
| example/multi-workflow/*.yaml | Four example files demonstrating multi-workflow chat platform with cross-workflow calls |
| module/*_test.go | Removes trailing blank lines (style cleanup) |
| for k, v := range wfCfg.Pipelines { | ||
| combined.Pipelines[k] = v |
There was a problem hiding this comment.
Pipeline name conflicts are not detected when merging multiple workflow files. If two workflow files define pipelines with the same name, the second will silently overwrite the first (line 214 uses simple map assignment). This could lead to unexpected behavior where one workflow's pipeline is inaccessible. Consider adding conflict detection similar to the module name conflict check on lines 198-204.
| for k, v := range wfCfg.Workflows { | ||
| combined.Workflows[k] = v | ||
| } | ||
| for k, v := range wfCfg.Triggers { | ||
| combined.Triggers[k] = v |
There was a problem hiding this comment.
Trigger name conflicts are not detected when merging multiple workflow files. Similar to the pipeline name conflict issue, if two workflow files define triggers with the same name, the second will silently overwrite the first (line 211 uses simple map assignment). Consider adding conflict detection for both triggers and workflows maps to match the module conflict detection on lines 198-204.
| if combined.ConfigDir == "" { | ||
| combined.ConfigDir = wfCfg.ConfigDir | ||
| } |
There was a problem hiding this comment.
ConfigDir assignment uses only the first workflow file's directory (lines 216-218). When merging multiple workflow files from different directories, this may cause issues if module configs reference relative paths - only paths relative to the first workflow's directory will resolve correctly. Consider either: (1) using the application config's directory consistently, or (2) preserving each module's original config directory in a per-module field, or (3) documenting that all relative paths must be relative to the first workflow file's directory.
|
@copilot open a new pull request to apply changes based on the comments in this thread |
…fig, async context propagation, comment accuracy (#145) * Initial plan * fix: add conflict detection for triggers/pipelines, fix async context, clarify comment Co-authored-by: intel352 <77607+intel352@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
| // applies the admin config overlay, then builds the engine using | ||
| // BuildFromApplicationConfig so cross-workflow pipeline calls are wired up. |
There was a problem hiding this comment.
The comment says the engine is built using BuildFromApplicationConfig, but the implementation merges configs and then calls buildEngine(combined, ...) (which uses BuildFromConfig). Either update the comment to match the actual flow, or switch to calling engine.BuildFromApplicationConfig(appCfg) if that’s the intended entry point so the code and documentation stay aligned.
| // applies the admin config overlay, then builds the engine using | |
| // BuildFromApplicationConfig so cross-workflow pipeline calls are wired up. | |
| // applies the admin config overlay, then builds the engine from the merged | |
| // configuration so cross-workflow pipeline calls are wired up. |
| cs := &inputCapturingStep{name: "capture"} | ||
| p := &Pipeline{Name: name, Steps: []PipelineStep{cs}} | ||
| // After Execute, copy step's captured data to the output pointer. | ||
| _ = captured |
There was a problem hiding this comment.
capturingPipeline has a redundant _ = captured assignment even though captured is used later when constructing triggerDataCapture. This can be removed to reduce noise and avoid suggesting the parameter is unused.
| _ = captured |
| // Register the PipelineWorkflowHandler (normally done via LoadPlugin) | ||
| engine.RegisterWorkflowHandler(handlers.NewPipelineWorkflowHandler()) |
There was a problem hiding this comment.
newTestEngine loads the pipelinesteps plugin (which already registers PipelineWorkflowHandler via WorkflowHandlers()), but then manually registers another PipelineWorkflowHandler and the comment says this is “normally done via LoadPlugin”. This duplicates handlers and makes the test setup diverge from production wiring. Consider removing the manual RegisterWorkflowHandler(...) (or adjust the comment/setup if you intentionally need a custom handler instance).
| // Register the PipelineWorkflowHandler (normally done via LoadPlugin) | |
| engine.RegisterWorkflowHandler(handlers.NewPipelineWorkflowHandler()) |
| if m, ok := cfg["mode"].(string); ok && m == string(WorkflowCallModeAsync) { | ||
| mode = WorkflowCallModeAsync |
There was a problem hiding this comment.
mode is only checked for the exact string "async" and any other (including typos like "asnyc") silently falls back to sync. This makes config errors hard to detect. Consider validating mode explicitly (accept only "sync"/"async") and returning a clear factory error for unknown values.
| if m, ok := cfg["mode"].(string); ok && m == string(WorkflowCallModeAsync) { | |
| mode = WorkflowCallModeAsync | |
| if rawMode, ok := cfg["mode"]; ok { | |
| m, ok := rawMode.(string) | |
| if !ok { | |
| return nil, fmt.Errorf("workflow_call step %q: 'mode' must be a string ('sync' or 'async')", name) | |
| } | |
| switch m { | |
| case "", string(WorkflowCallModeSync): | |
| mode = WorkflowCallModeSync | |
| case string(WorkflowCallModeAsync): | |
| mode = WorkflowCallModeAsync | |
| default: | |
| return nil, fmt.Errorf("workflow_call step %q: invalid 'mode' %q (must be 'sync' or 'async')", name, m) | |
| } |
Summary
ApplicationConfigformat for referencing multiple workflow YAML files in a single applicationstep.workflow_callpipeline step for cross-workflow invocation (sync and async modes)pipelineRegistryto StdEngine for runtime pipeline lookupBuildFromApplicationConfig()with module name conflict detectionTest plan
go build ./...compiles cleanlygo test ./...— all tests pass (including 482 lines of new tests)golangci-lint run— 0 issuesCloses #80
🤖 Generated with Claude Code