docs: comprehensive documentation overhaul

intel352 · claude · intel352 · commit 39e1b5a68886 · 2026-03-03T20:54:22.000-05:00
- README.md: add pkg.go.dev badge, mermaid architecture diagram, wfctl
  install instructions, library usage section, documentation index
- CLAUDE.md: new file with build/test commands, project structure,
  coding conventions, and documentation maintenance rules
- docs/WFCTL.md: new comprehensive CLI reference for all 22 wfctl
  commands with flags, examples, and mermaid command tree diagram
- DOCUMENTATION.md: update template functions from 9 to 30+ (add
  v0.3.9/v0.3.10 functions: upper, title, replace, contains, hasPrefix,
  hasSuffix, split, join, trimSpace, urlEncode, add, sub, mul, div,
  toInt, toFloat, toString, length, coalesce, config), add pipeline
  execution flow diagram, document step.hash and step.regex_match

Co-Authored-By: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,97 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Configuration-driven workflow orchestration engine built in Go on the [modular](https://github.com/CrisisTextLine/modular) framework. Turns YAML configs into running applications — API servers, event pipelines, state machines, and more.
+
+## Build & Run
+
+```sh
+# Build the server
+go build -o server ./cmd/server
+./server -config example/api-server-config.yaml
+
+# Build the CLI
+go build -o wfctl ./cmd/wfctl
+./wfctl validate example/api-server-config.yaml
+
+# Run tests
+go test ./...
+go test -race ./...
+
+# UI development
+cd ui && npm install && npm run dev
+
+# Lint
+go fmt ./...
+golangci-lint run
+```
+
+## Project Structure
+
+- `cmd/server/` — Server binary entry point
+- `cmd/wfctl/` — CLI tool (validate, inspect, deploy, api extract, etc.)
+- `config/` — YAML config structs
+- `module/` — All module and pipeline step implementations
+- `handlers/` — Workflow handler types (HTTP, Messaging, StateMachine, Scheduler, Integration)
+- `plugins/` — Built-in engine plugins (auth, storage, pipeline-steps, etc.)
+- `plugin/` — Plugin SDK (EnginePlugin interface, factories, hooks)
+- `schema/` — JSON Schema generation for config validation
+- `dynamic/` — Yaegi hot-reload system
+- `ai/` — AI integration (Anthropic Claude, GitHub Copilot)
+- `ui/` — React + ReactFlow visual builder (Vite, TypeScript)
+- `example/` — Example YAML configs with companion .md docs
+- `deploy/` — Deployment configs (Docker, Kubernetes/Helm, OpenTofu/AWS)
+- `docs/` — Documentation (tutorials, ADRs, API reference)
+
+## Key Conventions
+
+- Module implementations live in `module/` with the naming pattern `module/<type>.go`
+- Pipeline steps follow `module/pipeline_step_<name>.go`
+- Template functions are registered in `module/pipeline_template.go` via `templateFuncMap()`
+- Module factories are registered in `engine.go` `BuildFromConfig()`
+- Plugin step/module factories are registered in `plugins/<name>/plugin.go`
+- wfctl commands each have their own file in `cmd/wfctl/<command>.go`
+- Tests follow standard Go conventions: `*_test.go` alongside source files
+
+## Documentation Maintenance
+
+**When adding new functionality, update the corresponding documentation:**
+
+| Change | Update |
+|--------|--------|
+| New module type | `DOCUMENTATION.md` module table, `engine.go` factory registration |
+| New pipeline step | `DOCUMENTATION.md` step table, register in plugin or engine |
+| New template function | `DOCUMENTATION.md` template functions section, add to `templateFuncMap()` |
+| New wfctl command | `docs/WFCTL.md` command reference, update `main.go` usage() |
+| New trigger type | `DOCUMENTATION.md` trigger types section |
+| Config format change | `DOCUMENTATION.md` configuration section |
+
+## Common Tasks
+
+### Adding a New Module Type
+1. Create `module/<type>.go` implementing `modular.Module`
+2. Register factory in `engine.go` `BuildFromConfig()` or in a plugin's `ModuleFactories()`
+3. Add schema in the plugin's `ModuleSchemas()` or inline
+4. Add example YAML in `example/`
+5. Update `DOCUMENTATION.md`
+
+### Adding a New Pipeline Step
+1. Create `module/pipeline_step_<name>.go` implementing the step interface
+2. Register in `plugins/pipelinesteps/plugin.go` `StepFactories()`
+3. Update `DOCUMENTATION.md`
+
+### Adding a Template Function
+1. Add to `templateFuncMap()` in `module/pipeline_template.go`
+2. Add tests in `module/pipeline_template_test.go`
+3. Update `DOCUMENTATION.md` template functions section
+
+## Links
+
+- [Full Documentation](DOCUMENTATION.md)
+- [CLI Reference](docs/WFCTL.md)
+- [Deployment Guide](deploy/README.md)
+- [Tutorials](docs/tutorials/)
+- [Go Package Docs](https://pkg.go.dev/github.com/GoCodeAlone/workflow)
diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md
@@ -2,7 +2,7 @@
 
 ## Overview
 
-The Workflow Engine is a configuration-driven orchestration platform built in Go. It turns YAML configuration files into running applications with no code changes required. The engine provides 48+ built-in module types, a visual workflow builder UI, a multi-tenant admin platform, AI-assisted configuration generation, and dynamic hot-reload of Go components at runtime.
+The Workflow Engine is a configuration-driven orchestration platform built in Go. It turns YAML configuration files into running applications with no code changes required. The engine provides 50+ built-in module types, a visual workflow builder UI, a multi-tenant admin platform, AI-assisted configuration generation, and dynamic hot-reload of Go components at runtime.
 
 ## Core Engine
 
@@ -11,15 +11,15 @@ The engine is built on the [CrisisTextLine/modular](https://github.com/CrisisTex
 **Key capabilities:**
 - YAML-driven configuration with environment variable expansion (`${JWT_SECRET}`)
 - Config validation via JSON Schema
-- Module factory registry with 48 built-in types
+- Module factory registry with 50+ built-in types
 - Trigger-based workflow dispatch (HTTP, EventBus, cron schedule)
 - Graceful lifecycle management (start/stop)
 
 **CLI tools:**
 - `cmd/server` -- runs workflow configs as a server process
 - `cmd/wfctl` -- validates and inspects workflow configs offline
 
-## Module Types (48+)
+## Module Types (50+)
 
 All modules are registered in `engine.go` and instantiated from YAML config. Organized by category:
 
@@ -85,6 +85,28 @@ All modules are registered in `engine.go` and instantiated from YAML config. Org
 | `persistence.store` | Write-through persistence (SQLite/PostgreSQL) |
 
 ### Pipeline Steps
+
+Pipeline execution flow:
+
+```mermaid
+flowchart TD
+    A[HTTP Request] --> B[Trigger]
+    B --> C[WorkflowHandler]
+    C --> D[PipelineContext\ncurrent / steps / trigger / meta]
+    D --> E[Step 1\nresolve templates]
+    E --> F{StepResult}
+    F --> G[Merge output\ninto context]
+    G --> H[Step 2\nresolve templates]
+    H --> I{StepResult}
+    I --> J[Merge output\ninto context]
+    J --> K[... more steps ...]
+    K --> L[Final Response]
+
+    style D fill:#f0f4ff,stroke:#4a6fa5
+    style F fill:#e8f5e9,stroke:#388e3c
+    style I fill:#e8f5e9,stroke:#388e3c
+```
+
 | Type | Description |
 |------|-------------|
 | `processing.step` | Configurable processing step |
@@ -94,12 +116,42 @@ All modules are registered in `engine.go` and instantiated from YAML config. Org
 | `step.set` | Sets values in pipeline context with template support |
 | `step.log` | Logs pipeline data for debugging |
 | `step.publish` | Publishes events to EventBus |
+| `step.event_publish` | Publishes events to EventBus with full envelope control |
 | `step.http_call` | Makes outbound HTTP requests |
 | `step.delegate` | Delegates to a named service |
 | `step.request_parse` | Extracts path params, query params, and request body from HTTP requests |
 | `step.db_query` | Executes parameterized SQL SELECT queries against a named database |
 | `step.db_exec` | Executes parameterized SQL INSERT/UPDATE/DELETE against a named database |
+| `step.db_query_cached` | Executes a cached SQL SELECT query |
+| `step.db_create_partition` | Creates a time-based table partition |
+| `step.db_sync_partitions` | Ensures future partitions exist for a partitioned table |
 | `step.json_response` | Writes HTTP JSON response with custom status code and headers |
+| `step.raw_response` | Writes a raw HTTP response with arbitrary content type |
+| `step.static_file` | Serves a pre-loaded file from disk as an HTTP response |
+| `step.workflow_call` | Invokes another workflow pipeline by name |
+| `step.validate_path_param` | Validates a URL path parameter against a set of rules |
+| `step.validate_pagination` | Validates and normalizes pagination query params |
+| `step.validate_request_body` | Validates request body against a JSON schema |
+| `step.foreach` | Iterates over a slice and runs a sub-pipeline per element |
+| `step.webhook_verify` | Verifies an inbound webhook signature |
+| `step.base64_decode` | Decodes a base64-encoded field |
+| `step.cache_get` | Reads a value from the cache module |
+| `step.cache_set` | Writes a value to the cache module |
+| `step.cache_delete` | Deletes a value from the cache module |
+| `step.ui_scaffold` | Generates UI scaffolding from a workflow config |
+| `step.ui_scaffold_analyze` | Analyzes UI scaffold state for a workflow |
+| `step.dlq_send` | Sends a message to the dead-letter queue |
+| `step.dlq_replay` | Replays messages from the dead-letter queue |
+| `step.retry_with_backoff` | Retries a sub-pipeline with exponential backoff |
+| `step.resilient_circuit_breaker` | Wraps a sub-pipeline with a circuit breaker |
+| `step.s3_upload` | Uploads a file or data to an S3-compatible bucket |
+| `step.auth_validate` | Validates an authentication token and populates claims |
+| `step.token_revoke` | Revokes an auth token |
+| `step.field_reencrypt` | Re-encrypts a field with a new key |
+| `step.sandbox_exec` | Executes a command inside a sandboxed container |
+| `step.http_proxy` | Proxies an HTTP request to an upstream service |
+| `step.hash` | Computes a cryptographic hash (md5/sha256/sha512) of a template-resolved input |
+| `step.regex_match` | Matches a regular expression against a template-resolved input |
 | `step.jq` | Applies a JQ expression to pipeline data for complex transformations |
 | `step.ai_complete` | AI text completion using a configured provider |
 | `step.ai_classify` | AI text classification into named categories |
@@ -121,17 +173,60 @@ All modules are registered in `engine.go` and instantiated from YAML config. Org
 
 Pipeline steps support Go template syntax with these built-in functions:
 
-| Function | Description | Example |
-|----------|-------------|---------|
-| `uuidv4` | Generates a UUID v4 | `{{ uuidv4 }}` |
-| `now` | Current time (RFC3339 default, or named/custom layout) | `{{ now }}`, `{{ now "DateOnly" }}`, `{{ now "2006-01-02" }}` |
-| `lower` | Lowercase string | `{{ lower .name }}` |
-| `default` | Default value when empty | `{{ default "pending" .status }}` |
-| `json` | Marshal value to JSON string | `{{ json .data }}` |
-| `trimPrefix` | Remove prefix from string | `{{ .phone | trimPrefix "+" }}` |
-| `trimSuffix` | Remove suffix from string | `{{ .file | trimSuffix ".txt" }}` |
-| `step` | Access step outputs by name and keys | `{{ step "parse-request" "path_params" "id" }}` |
-| `trigger` | Access trigger data by keys | `{{ trigger "path_params" "id" }}` |
+#### Core
+
+| Function | Signature | Description |
+|----------|-----------|-------------|
+| `uuid` | `uuid` | Generate UUID v4 |
+| `uuidv4` | `uuidv4` | Generate UUID v4 (alias for `uuid`) |
+| `now` | `now [layout]` | Current UTC time (default RFC3339); accepts named constants (`RFC3339`, `DateOnly`, etc.) or a Go layout string |
+| `lower` | `lower STRING` | Lowercase |
+| `default` | `default FALLBACK VALUE` | Return fallback if value is nil/empty |
+| `json` | `json VALUE` | Marshal to JSON string |
+| `config` | `config KEY` | Look up a value from the config registry (populated by a `config.provider` module) |
+
+#### String
+
+| Function | Signature | Description |
+|----------|-----------|-------------|
+| `upper` | `upper STRING` | Uppercase |
+| `title` | `title STRING` | Title case (first letter of each word capitalized) |
+| `replace` | `replace OLD NEW STRING` | Replace all occurrences of OLD with NEW |
+| `contains` | `contains SUBSTR STRING` | Check if STRING contains SUBSTR |
+| `hasPrefix` | `hasPrefix PREFIX STRING` | Check if STRING starts with PREFIX |
+| `hasSuffix` | `hasSuffix SUFFIX STRING` | Check if STRING ends with SUFFIX |
+| `split` | `split SEP STRING` | Split STRING by SEP into a slice |
+| `join` | `join SEP SLICE` | Join slice elements with SEP |
+| `trimSpace` | `trimSpace STRING` | Trim leading and trailing whitespace |
+| `trimPrefix` | `trimPrefix PREFIX STRING` | Remove PREFIX from STRING if present |
+| `trimSuffix` | `trimSuffix SUFFIX STRING` | Remove SUFFIX from STRING if present |
+| `urlEncode` | `urlEncode STRING` | URL percent-encode a string |
+
+#### Math
+
+| Function | Signature | Description |
+|----------|-----------|-------------|
+| `add` | `add A B` | Addition (int if both ints, float64 otherwise) |
+| `sub` | `sub A B` | Subtraction |
+| `mul` | `mul A B` | Multiplication |
+| `div` | `div A B` | Division as float64; returns 0 on divide-by-zero |
+
+#### Type / Utility
+
+| Function | Signature | Description |
+|----------|-----------|-------------|
+| `toInt` | `toInt VALUE` | Convert to int64 |
+| `toFloat` | `toFloat VALUE` | Convert to float64 |
+| `toString` | `toString VALUE` | Convert to string |
+| `length` | `length VALUE` | Length of string, slice, array, or map |
+| `coalesce` | `coalesce VAL1 VAL2 ...` | First non-nil, non-empty value |
+
+#### Context (added per-pipeline by the engine)
+
+| Function | Signature | Description |
+|----------|-----------|-------------|
+| `step` | `step NAME KEY...` | Access a prior step's output by step name and nested keys |
+| `trigger` | `trigger KEY...` | Access trigger data by keys |
 
 #### Template Data Context
 
diff --git a/README.md b/README.md
@@ -3,6 +3,7 @@
 [![Go](https://img.shields.io/badge/Go-1.26+-00ADD8?logo=go&logoColor=white)](https://go.dev)
 [![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![Built on Modular](https://img.shields.io/badge/Built%20on-CrisisTextLine%2Fmodular-green)](https://github.com/CrisisTextLine/modular)
+[![Go Reference](https://pkg.go.dev/badge/github.com/GoCodeAlone/workflow.svg)](https://pkg.go.dev/github.com/GoCodeAlone/workflow)
 
 A production-grade, configuration-driven workflow orchestration engine built on [CrisisTextLine/modular](https://github.com/CrisisTextLine/modular) v1.11.11. Define entire applications in YAML -- from API servers to multi-service chat platforms -- with 48+ module types, dynamic hot-reload, AI-powered generation, and a visual builder UI.
 
@@ -161,6 +162,40 @@ cd example/ecommerce-app
 docker compose up
 ```
 
+## CLI Tool (wfctl)
+
+`wfctl` is a command-line tool for inspecting, validating, and generating workflow configurations.
+
+**Install from GitHub Releases:**
+```bash
+curl -sL https://github.com/GoCodeAlone/workflow/releases/latest/download/wfctl-$(uname -s | tr A-Z a-z)-$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') -o wfctl && chmod +x wfctl
+```
+
+**Install from source:**
+```bash
+go install github.com/GoCodeAlone/workflow/cmd/wfctl@latest
+```
+
+**Commands:**
+- `wfctl inspect <config.yaml>` — summarize modules, workflows, triggers, and dependencies
+- `wfctl validate <config.yaml>` — deep validation against known module/step types
+- `wfctl api-extract <config.yaml>` — generate OpenAPI 3.0 spec from HTTP workflows
+- `wfctl diff <old.yaml> <new.yaml>` — compare configs and detect breaking changes
+- `wfctl manifest <config.yaml>` — produce infrastructure requirements manifest
+- `wfctl scaffold --modules <types>` — generate a skeleton config for given module types
+
+See [docs/WFCTL.md](docs/WFCTL.md) for the full CLI reference.
+
+## Library Usage
+
+Use the workflow engine as a Go library:
+
+```go
+go get github.com/GoCodeAlone/workflow@latest
+```
+
+See the [Go package documentation](https://pkg.go.dev/github.com/GoCodeAlone/workflow) for API reference.
+
 ## Example Applications
 
 ### Chat Platform -- Production-Grade Mental Health Support
@@ -208,23 +243,30 @@ Each example includes a companion `.md` file documenting its architecture and us
 
 ## Architecture
 
-```
-cmd/server/          Server binary, HTTP mux, graceful shutdown
-  main.go            Entry point with CLI flags and AI provider init
-
-config/              YAML config structs (WorkflowConfig, ModuleConfig)
-module/              65+ built-in module implementations
-handlers/            5 workflow handler types:
-                       HTTP, Messaging, StateMachine, Scheduler, Integration
-dynamic/             Yaegi-based hot-reload system
-ai/                  AI integration layer
-  llm/                 Anthropic Claude direct API with tool use
-  copilot/             GitHub Copilot SDK with session management
-  service.go           Provider selection and orchestration
-  deploy.go            Validation loop and deployment to dynamic components
-ui/                  React + ReactFlow + Zustand visual builder (Vite, TypeScript)
-example/             Top-level example YAML configs and full application examples
-mock/                Test helpers and mock implementations
+```mermaid
+graph TD
+    A[YAML Config] --> B[StdEngine / BuildFromConfig]
+    B --> C[Module Factories]
+    C --> D[Modules]
+    D --> E[Application\nDependency Injection]
+
+    E --> F[Workflow Handlers]
+    F --> F1[HTTP]
+    F --> F2[Messaging]
+    F --> F3[StateMachine]
+    F --> F4[Scheduler]
+    F --> F5[Integration]
+
+    G[Triggers] --> G1[HTTP Endpoints]
+    G --> G2[EventBus Subscriptions]
+    G --> G3[Cron Schedules]
+
+    G1 --> H[TriggerWorkflow]
+    G2 --> H
+    G3 --> H
+    H --> I[Handler Dispatch]
+    I --> J[Pipeline Execution]
+    J --> J1[Steps: validate, transform,\nconditional, http_call, db_query...]
 ```
 
 **Core flow:**
@@ -299,6 +341,13 @@ cd ui && npm run lint
 | Containers | Docker multi-stage builds, Docker Compose |
 | Testing | Go testing, Vitest, Playwright |
 
+## Documentation
+
+- [DOCUMENTATION.md](DOCUMENTATION.md) — Full module, step, and template reference
+- [docs/WFCTL.md](docs/WFCTL.md) — CLI tool reference
+- [deploy/README.md](deploy/README.md) — Deployment guide
+- [docs/tutorials/](docs/tutorials/) — Getting started and walkthrough tutorials
+
 ## Roadmap
 
 See [ROADMAP.md](ROADMAP.md) for the full development history (Phases 1-6 complete) and planned work including JSON Schema config validation, performance benchmarks, Helm charts, and security hardening.
diff --git a/docs/WFCTL.md b/docs/WFCTL.md
