Skip to content

chore: migrate workflows to db #818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adityachoudhari26 merged 5 commits into from
Feb 26, 2026
Merged

chore: migrate workflows to db #818

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9d3ebe4
chore: migrate workflows to db
adityachoudhari26 Feb 26, 2026
e559418
more fix
adityachoudhari26 Feb 26, 2026
474c596
cleanup
adityachoudhari26 Feb 26, 2026
3cf05de
cleanup
adityachoudhari26 Feb 26, 2026
93149a0
fmt
adityachoudhari26 Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion apps/workspace-engine/oapi/openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2672,13 +2672,17 @@
"ref": {
"description": "Reference to the job agent",
"type": "string"
},
"workflowId": {
"type": "string"
}
},
"required": [
"id",
"name",
"ref",
"config"
"config",
"workflowId"
],
"type": "object"
},
Expand Down
3 changes: 2 additions & 1 deletion apps/workspace-engine/oapi/spec/schemas/workflows.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,11 @@ local openapi = import '../lib/openapi.libsonnet';

WorkflowJobTemplate: {
type: 'object',
required: ['id', 'name', 'ref', 'config'],
required: ['id', 'name', 'ref', 'config', 'workflowId'],
properties: {
name: { type: 'string' },
id: { type: 'string' },
workflowId: { type: 'string' },
ref: { type: 'string', description: 'Reference to the job agent' },
config: { type: 'object', additionalProperties: true, description: 'Configuration for the job agent' },
matrix: openapi.schemaRef('WorkflowJobMatrix'),
Expand Down
32 changes: 32 additions & 0 deletions apps/workspace-engine/pkg/db/models.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 32 additions & 0 deletions apps/workspace-engine/pkg/db/queries/schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,38 @@ CREATE TABLE deployment_variable_value (
priority BIGINT NOT NULL DEFAULT 0
);

CREATE TABLE workflow (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
inputs JSONB NOT NULL DEFAULT '[]',
jobs JSONB NOT NULL DEFAULT '[]',
workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE
);

CREATE TABLE workflow_job_template (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
name TEXT NOT NULL,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
if_condition TEXT,
matrix JSONB
);

CREATE TABLE workflow_run (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
inputs JSONB NOT NULL DEFAULT '{}'
);

CREATE TABLE workflow_job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_run_id UUID NOT NULL REFERENCES workflow_run(id) ON DELETE CASCADE,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
index INTEGER NOT NULL DEFAULT 0
);
Comment on lines +252 to +282
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add indexes on workflow foreign keys in SQLC schema.

These FK columns should be indexed to keep list/query paths and cascading cleanup performant.

📌 Suggested SQL additions 
 CREATE TABLE workflow_job (
     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
     workflow_run_id UUID NOT NULL REFERENCES workflow_run(id) ON DELETE CASCADE,
     ref TEXT NOT NULL,
     config JSONB NOT NULL DEFAULT '{}',
     index INTEGER NOT NULL DEFAULT 0
 );
+
+CREATE INDEX workflow_workspace_id_idx ON workflow(workspace_id);
+CREATE INDEX workflow_job_template_workflow_id_idx ON workflow_job_template(workflow_id);
+CREATE INDEX workflow_run_workflow_id_idx ON workflow_run(workflow_id);
+CREATE INDEX workflow_job_workflow_run_id_idx ON workflow_job(workflow_run_id);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
CREATE TABLE workflow (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
inputs JSONB NOT NULL DEFAULT '[]',
jobs JSONB NOT NULL DEFAULT '[]',
workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE
);
CREATE TABLE workflow_job_template (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
name TEXT NOT NULL,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
if_condition TEXT,
matrix JSONB
);
CREATE TABLE workflow_run (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
inputs JSONB NOT NULL DEFAULT '{}'
);
CREATE TABLE workflow_job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_run_id UUID NOT NULL REFERENCES workflow_run(id) ON DELETE CASCADE,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
index INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE workflow (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
inputs JSONB NOT NULL DEFAULT '[]',
jobs JSONB NOT NULL DEFAULT '[]',
workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE
);
CREATE TABLE workflow_job_template (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
name TEXT NOT NULL,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
if_condition TEXT,
matrix JSONB
);
CREATE TABLE workflow_run (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflow(id) ON DELETE CASCADE,
inputs JSONB NOT NULL DEFAULT '{}'
);
CREATE TABLE workflow_job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_run_id UUID NOT NULL REFERENCES workflow_run(id) ON DELETE CASCADE,
ref TEXT NOT NULL,
config JSONB NOT NULL DEFAULT '{}',
index INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX workflow_workspace_id_idx ON workflow(workspace_id);
CREATE INDEX workflow_job_template_workflow_id_idx ON workflow_job_template(workflow_id);
CREATE INDEX workflow_run_workflow_id_idx ON workflow_run(workflow_id);
CREATE INDEX workflow_job_workflow_run_id_idx ON workflow_job(workflow_run_id);
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@apps/workspace-engine/pkg/db/queries/schema.sql` around lines 252 - 282, The
foreign key columns lack indexes which will hurt list/query and cascade
performance; add explicit B-tree indexes on workflow.workspace_id,
workflow_job_template.workflow_id, workflow_run.workflow_id, and
workflow_job.workflow_run_id (i.e., create indexes for each FK column) so
lookups and cascading deletes are efficient and the SQLC schema reflects those
indexes.

Copy link
Member Author

@adityachoudhari26 adityachoudhari26 Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

premature optimiziation

Copy link
Contributor

@coderabbitai coderabbitai bot Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adityachoudhari26, understood! You can always add these indexes later if query performance becomes a concern.

✏️ Learnings added 
Learnt from: adityachoudhari26
Repo: ctrlplanedev/ctrlplane PR: 818
File: apps/workspace-engine/pkg/db/queries/schema.sql:252-282
Timestamp: 2026-02-26T23:01:19.319Z
Learning: User adityachoudhari26 prefers to defer adding database indexes on foreign keys until there's evidence they're needed, considering such additions premature optimization at the initial schema design stage.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

🧠 Learnings used 
Learnt from: adityachoudhari26
Repo: ctrlplanedev/ctrlplane PR: 668
File: apps/webservice/src/app/api/v1/deployments/[deploymentId]/variables/variable-diff-check.ts:104-134
Timestamp: 2025-09-26T01:53:05.472Z
Learning: The deployment variable value tables in the ctrlplane/db schema do not have timestamp fields like createdAt or updatedAt. The deploymentVariableValueReference table has fields: id, variableValueId, reference, path, defaultValue. The deploymentVariableValue table has fields: id, variableId, resourceSelector, priority.

Learnt from: adityachoudhari26
Repo: ctrlplanedev/ctrlplane PR: 668
File: apps/webservice/src/app/api/v1/deployments/[deploymentId]/variables/route.ts:185-194
Timestamp: 2025-09-26T01:51:59.096Z
Learning: The deploymentVariable schema in ctrlplane/db does not contain timestamp fields like createdAt or updatedAt - it only has id, key, description, deploymentId, defaultValueId, and config fields.

Learnt from: adityachoudhari26
Repo: ctrlplanedev/ctrlplane PR: 637
File: packages/events/src/kafka/client.ts:10-16
Timestamp: 2025-08-01T04:41:41.345Z
Learning: User adityachoudhari26 prefers not to add null safety checks for required environment variables when they are guaranteed to be present in their deployment configuration, similar to their preference for simplicity over defensive programming in test code.

Learnt from: adityachoudhari26
Repo: ctrlplanedev/ctrlplane PR: 601
File: e2e/tests/api/policies/retry-policy.spec.ts:23-24
Timestamp: 2025-06-24T23:52:50.732Z
Learning: The user adityachoudhari26 prefers not to add null safety checks or defensive programming in test code, particularly in e2e tests, as they prioritize simplicity and focus on the main functionality being tested rather than comprehensive error handling within the test itself.


CREATE TABLE resource_variable (
resource_id UUID NOT NULL REFERENCES resource(id) ON DELETE CASCADE,
key TEXT NOT NULL,
Expand Down
72 changes: 72 additions & 0 deletions apps/workspace-engine/pkg/db/queries/workflows.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
-- name: GetWorkflowByID :one
SELECT id, name, inputs, jobs, workspace_id FROM workflow WHERE id = $1;

-- name: ListWorkflowsByWorkspaceID :many
SELECT id, name, inputs, jobs, workspace_id FROM workflow WHERE workspace_id = $1;

-- name: UpsertWorkflow :one
INSERT INTO workflow (id, name, inputs, jobs, workspace_id) VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, inputs = EXCLUDED.inputs, jobs = EXCLUDED.jobs
RETURNING *;

-- name: DeleteWorkflow :exec
DELETE FROM workflow WHERE id = $1;

-- name: GetWorkflowJobTemplateByID :one
SELECT id, workflow_id, name, ref, config, if_condition, matrix FROM workflow_job_template WHERE id = $1;

-- name: ListWorkflowJobTemplatesByWorkspaceID :many
SELECT wjt.id, wjt.workflow_id, wjt.name, wjt.ref, wjt.config, wjt.if_condition, wjt.matrix
FROM workflow_job_template wjt
INNER JOIN workflow w ON w.id = wjt.workflow_id
WHERE w.workspace_id = $1;

-- name: UpsertWorkflowJobTemplate :one
INSERT INTO workflow_job_template (id, workflow_id, name, ref, config, if_condition, matrix) VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (id) DO UPDATE SET workflow_id = EXCLUDED.workflow_id, name = EXCLUDED.name, ref = EXCLUDED.ref, config = EXCLUDED.config, if_condition = EXCLUDED.if_condition, matrix = EXCLUDED.matrix
RETURNING *;

-- name: DeleteWorkflowJobTemplate :exec
DELETE FROM workflow_job_template WHERE id = $1;

-- name: GetWorkflowRunByID :one
SELECT id, workflow_id, inputs FROM workflow_run WHERE id = $1;

-- name: ListWorkflowRunsByWorkflowID :many
SELECT id, workflow_id, inputs FROM workflow_run WHERE workflow_id = $1;

-- name: ListWorkflowRunsByWorkspaceID :many
SELECT wr.id, wr.workflow_id, wr.inputs
FROM workflow_run wr
INNER JOIN workflow w ON w.id = wr.workflow_id
WHERE w.workspace_id = $1;

-- name: UpsertWorkflowRun :one
INSERT INTO workflow_run (id, workflow_id, inputs) VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE SET workflow_id = EXCLUDED.workflow_id, inputs = EXCLUDED.inputs
RETURNING *;

-- name: DeleteWorkflowRun :exec
DELETE FROM workflow_run WHERE id = $1;

-- name: GetWorkflowJobByID :one
SELECT id, workflow_run_id, ref, config, index FROM workflow_job WHERE id = $1;

-- name: ListWorkflowJobsByWorkflowRunID :many
SELECT id, workflow_run_id, ref, config, index FROM workflow_job WHERE workflow_run_id = $1 ORDER BY index ASC;

-- name: ListWorkflowJobsByWorkspaceID :many
SELECT wj.id, wj.workflow_run_id, wj.ref, wj.config, wj.index
FROM workflow_job wj
INNER JOIN workflow_run wr ON wr.id = wj.workflow_run_id
INNER JOIN workflow w ON w.id = wr.workflow_id
WHERE w.workspace_id = $1
ORDER BY wj.index ASC;

-- name: UpsertWorkflowJob :one
INSERT INTO workflow_job (id, workflow_run_id, ref, config, index) VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE SET workflow_run_id = EXCLUDED.workflow_run_id, ref = EXCLUDED.ref, config = EXCLUDED.config, index = EXCLUDED.index
RETURNING *;

-- name: DeleteWorkflowJob :exec
DELETE FROM workflow_job WHERE id = $1;
27 changes: 27 additions & 0 deletions apps/workspace-engine/pkg/db/sqlc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ sql:
- queries/user_approval_records.sql
- queries/resource_variables.sql
- queries/deployment_variables.sql
- queries/workflows.sql
database:
uri: "postgresql://ctrlplane:ctrlplane@127.0.0.1:5432/ctrlplane?sslmode=disable"
gen:
Expand Down Expand Up @@ -115,3 +116,29 @@ sql:
- column: "deployment_variable_value.value"
go_type:
type: "[]byte"

# Workflow
- column: "workflow.inputs"
go_type:
type: "[]byte"
- column: "workflow.jobs"
go_type:
type: "[]byte"

# WorkflowJobTemplate
- column: "workflow_job_template.config"
go_type:
type: "map[string]any"
- column: "workflow_job_template.matrix"
go_type:
type: "[]byte"

# WorkflowRun
- column: "workflow_run.inputs"
go_type:
type: "map[string]any"

# WorkflowJob
- column: "workflow_job.config"
go_type:
type: "map[string]any"
Loading
Loading