Skip to content

Pipeline API

Jump to bottom
Dmitrii Karataev edited this page Feb 26, 2026 · 2 revisions

Pipeline API

Pipelines are the core abstraction in RocketRide. A pipeline defines a directed graph of processing nodes and the data flow between them. This page documents the pipeline JSON format, services.json node metadata, and configuration mechanisms.

Pipeline JSON Structure

A pipeline file has three top-level sections:

{
    "version": "1.0",
    "components": [ ... ],
    "edges": [ ... ]
}

Components

Each component represents a pipeline node instance:

{
    "id": "my_llm",
    "type": "llm_openai://",
    "parameters": {
        "llm_openai": {
            "profile": "openai-5-2",
            "openai-5-2": {
                "apikey": "${OPENAI_API_KEY}"
            }
        }
    }
}
Field Required Description
id Yes Unique identifier for this component within the pipeline
type Yes Protocol URI identifying the node type (e.g., llm_openai://)
parameters Yes Node-specific configuration, structured according to the node's services.json

Edges

Edges connect components via lanes:

{
    "from": "embedder",
    "to": "store",
    "lane": "documents"
}
Field Required Description
from Yes Source component ID
to Yes Target component ID (use null for pipeline output)
lane Yes Data lane name (e.g., documents, questions, answers)

Lanes

Lanes are typed data channels. Each node declares which lanes it accepts and which it produces. See Data Flow for the full list.

Common lane patterns:

Pattern Input Lane Output Lane Use Case
Ingest text documents Preprocessing / chunking
Embed documents documents Adding vector embeddings
Store documents (none) Writing to vector DB
Search questions documents Similarity search
Answer questions answers LLM generation

services.json

Every pipeline node has a services.json file that defines its metadata, configuration schema, and lane declarations. This file lives alongside the node's Python code in nodes/src/nodes/<node_name>/.

Key Fields

{
    "title": "LLM - OpenAI",
    "protocol": "llm_openai://",
    "classType": ["llm"],
    "capabilities": ["invoke"],
    "register": "filter",
    "node": "python",
    "path": "nodes.llm_openai",
    "prefix": "llm",
    "description": ["..."],
    "icon": "openai.svg",
    "documentation": "https://...",
    "lanes": { ... },
    "input": [ ... ],
    "preconfig": { ... },
    "fields": { ... },
    "shape": [ ... ],
    "test": { ... }
}
Field Required Description
title Yes Display name shown in the UI
protocol Yes Unique protocol URI used as the component type
classType Yes Node classification (e.g., llm, store, embedding, preprocessor)
capabilities Yes Engine behavior flags (e.g., invoke)
register No Factory registration type: filter or endpoint
node No Runtime type (typically python)
path No Python module path for the node implementation
prefix Yes URL path prefix mapping
description No Array of strings describing the node
icon No SVG icon filename for the UI
documentation No Link to external documentation
lanes No Lane mapping: input lanes to output lanes
input No Detailed input/output lane descriptions
preconfig No Profile definitions and defaults
fields No Field definitions for the configuration UI
shape Yes UI form layout defining which fields appear in each section
test No Test configuration for the node test framework

Profiles and Preconfig

Nodes use profiles to offer pre-defined configurations. The preconfig section defines available profiles and their default values:

"preconfig": {
    "default": "openai-5-2",
    "profiles": {
        "openai-5-2": {
            "title": "GPT-5.2",
            "model": "gpt-5.2",
            "modelTotalTokens": 128000,
            "apikey": ""
        },
        "openai-4o": {
            "title": "GPT-4o",
            "model": "gpt-4o",
            "modelTotalTokens": 128000,
            "apikey": ""
        }
    }
}

In the pipeline JSON, select a profile and override specific values:

"parameters": {
    "llm_openai": {
        "profile": "openai-5-2",
        "openai-5-2": {
            "apikey": "${OPENAI_API_KEY}"
        }
    }
}

Environment Variable Substitution

Use ${VAR_NAME} syntax in pipeline JSON values. Variables are resolved at pipeline load time from:

  1. The env dictionary passed via the client SDK
  2. Server-side environment variables
  3. .env file in the working directory
"apikey": "${OPENAI_API_KEY}"

Test Configuration

Nodes can define test cases in their services.json for automated testing:

"test": {
    "profiles": ["openai-5-mini"],
    "outputs": ["answers"],
    "cases": [
        {
            "name": "LLM returns mock response",
            "text": "What is 2+2?",
            "expect": {
                "answers": {
                    "contains": "Mock LLM response"
                }
            }
        }
    ]
}

Run node tests with:

./builder nodes:test        # Contract tests (mocked)
./builder nodes:test-full   # Integration tests (requires running server)

Next Steps

Getting Started

Architecture

API Reference

Contributing

Governance

Clone this wiki locally