diff --git a/content/symbiotic/architecture.mdx b/content/symbiotic/architecture.mdx new file mode 100644 index 00000000..fa5993ec --- /dev/null +++ b/content/symbiotic/architecture.mdx @@ -0,0 +1,144 @@ +--- +title: Architecture +--- + +System overview for the Symbiotic multi-provider template. + +## Core Model + +1. One active provider per running stack (`config/environments/.json`). +2. Shared off-chain runtime: + - OZ Monitor for ingress + - 3 operator processes + - 3 Symbiotic relay sidecars for BLS signatures + - OZ Relayer for destination tx submission + - Redis queue +3. Provider-specific on-chain contracts and calldata format. + +## Provider Matrix + +| Provider | Source ingress event | Destination submit call | Local | Testnet | Mainnet | +| --- | --- | --- | --- | --- | --- | +| [`layerzero`](/symbiotic/layerzero) | `JobAssigned` | `SymbioticLayerZeroDVN.submitProof(...)` | Supported | Supported | Not yet | +| [`chainlink_ccv`](/symbiotic/chainlink-ccv) | `CCIPMessageSent` | `OffRamp.execute(...)` | Supported | Not yet | Not yet | + +See per-provider pages for detailed message flows and code pointers. + +## Shared Off-Chain Runtime + +```mermaid +flowchart LR + subgraph source["Source Chain"] + Event["Provider Ingress Event"] + end + + Event --> Monitor["OZ Monitor"] + Monitor --> Operators["Operators (x3)"] + Operators <--> Relays["Symbiotic Relays\n(BLS sidecars)"] + Operators --> Relayer["OZ Relayer"] + + subgraph dest["Destination Chain"] + Submit["Provider Destination Call"] + end + + Relayer --> Submit +``` + +All providers share the same off-chain pipeline. The provider abstraction determines: +- Which event the monitor watches for +- How operators encode the payload +- What calldata the relayer submits + +## Merkle Tree Batching + +Messages are batched into Merkle trees for gas efficiency: + +1. Multiple messages are collected into a batch +2. Each message becomes a leaf in the Merkle tree +3. The Merkle root is signed by operators +4. Proofs allow verifying individual messages against the signed root + +This means: +- One signature covers many messages +- On-chain verification cost is amortized +- Individual messages can be verified independently + +## Symbiotic Integration + +Symbiotic provides the shared security layer: + +- **Operator Registration**: Operators stake and register their BLS public keys +- **Settlement Contract**: Verifies BLS signatures and checks quorum +- **Slashing**: Misbehaving operators can be penalized (production) + +The Settlement contract: +1. Maintains the list of registered operators and their public keys +2. Defines the quorum threshold +3. Verifies aggregated signatures +4. Reports verification results to the provider contract + +## BLS Signing Pipeline + +1. Operators sign provider-defined payloads through Symbiotic relay sidecars. +2. Aggregation/quorum logic comes from settlement-backed Symbiotic attestation rules. +3. Provider-specific contracts decode and enforce those attestations on the destination execution path. + +## Message Status Lifecycle + +```mermaid +flowchart LR + Pending --> Processing --> Signed --> Submitted --> Confirmed +``` + +| Status | Description | +|--------|-------------| +| Pending | Received via webhook, awaiting batching | +| Processing | Batched into Merkle tree, awaiting BLS signatures | +| Signed | Quorum signatures collected, ready for submission | +| Submitted | Sent to OZ Relayer | +| Confirmed | On-chain TX confirmed | + +## Operator Internals + +| Module | Location | Purpose | +|--------|----------|---------| +| API Server | `operator/src/api/` | Axum HTTP server, webhook endpoint, debug routes | +| Provider | `operator/src/provider/` | Provider trait, event decoding, message storage | +| SignerJob | `operator/src/signer/` | Batches messages into Merkle trees, requests BLS signatures | +| RelaySubmitterJob | `operator/src/relay_submitter/` | Submits signed proofs via OZ Relayer | +| Storage | `operator/src/storage/` | redb key-value store (messages, Merkle trees, submissions) | +| Crypto | `operator/src/crypto/` | Merkle tree construction, leaf hashing, signing message encoding | + +## Adding a New Provider + +1. Create `operator/src/provider/yourprovider.rs` implementing the `Provider` trait: + +```rust +#[async_trait] +pub trait Provider: Send + Sync + 'static { + fn name(&self) -> &'static str; + async fn handle_webhook_event(&self, event: &WebhookEvent) -> Result<(), ProviderError>; + + // Optional overrides: + fn register_api_routes(&self, router: Router) -> Router { router } + async fn acceptance_hook(&self, _msg: &MessageData) -> Result<(), ProviderError> { Ok(()) } +} +``` + +2. Add configuration to `operator/src/config/mod.rs`. +3. Register in `create_provider()` in `operator/src/provider/mod.rs`. +4. Create provider-specific monitor template in `config/templates/oz-monitor/monitors/`. +5. Create `docs/.mdx` following the structure of existing provider docs (e.g., [LayerZero](/symbiotic/layerzero)). +6. Update this file's provider matrix, the [docs index](/symbiotic), and the project README. + +## Environment Comparison + +| Aspect | Local (Anvil) | Testnet | Production | +|--------|---------------|---------|------------| +| Source chain | Anvil 31337 | Base Sepolia 84532 | Mainnet | +| Dest/Settlement chain | Anvil 31338 | Sepolia 11155111 | Mainnet | +| Operators | 3 (local containers) | 3 (local containers) | 1+ (distributed) | +| Symbiotic Core | Deployed locally | Pre-deployed on Sepolia | Pre-deployed | +| BLS Keys | Deterministic | Deterministic | Hardware security | +| Quorum | 2-of-3 | 2-of-3 | Configurable | +| OZ Services | Local | Local | Hosted by OZ | diff --git a/content/symbiotic/chainlink-ccv.mdx b/content/symbiotic/chainlink-ccv.mdx new file mode 100644 index 00000000..1ccf9aad --- /dev/null +++ b/content/symbiotic/chainlink-ccv.mdx @@ -0,0 +1,131 @@ +--- +title: Chainlink CCV +--- + +Symbiotic-secured Cross-Chain Verifier (CCV) for Chainlink CCIP-compatible message verification. + +## Overview + +The CCV provider implements a Symbiotic-backed verifier compatible with Chainlink's CCIP CCV interface. When a message is sent through an OnRamp-compatible contract, a `CCIPMessageSent` event is emitted. Operators build a CCV payload, collect BLS attestations via Symbiotic relay sidecars, and the relayer submits the proof to `OffRamp.execute(...)` on the destination chain. The OffRamp calls `SymbioticCCV.verifyMessage(...)` for each message, and success is confirmed when `MessageExecuted(messageId)` is emitted. + + + +This template supports the **Symbiotic CCV variant** only. The Chainlink auxiliary devenv stack (aggregator, indexer, verifier, executor) is not required. + + + +## Message Flow + +```mermaid +sequenceDiagram + participant App as Source App + participant OnRamp as OnRamp (Source) + participant Monitor as OZ Monitor + participant Operators as Operators (x3) + participant Relay as Symbiotic Relay (BLS) + participant Relayer as OZ Relayer + participant OffRamp as OffRamp (Dest) + participant CCV as SymbioticCCV (Dest) + + App->>OnRamp: sendMessage() + OnRamp-->>Monitor: CCIPMessageSent event + Monitor->>Operators: HMAC webhook + Operators->>Operators: build CCV payload + Merkle tree + Operators->>Relay: sign Merkle root (BLS) + Relay-->>Operators: aggregated signature + Operators->>Relayer: OffRamp.execute calldata + Relayer->>OffRamp: execute(message, ccvs, verifierResults) + OffRamp->>CCV: verifyMessage(message, messageId, verifierResults) + CCV-->>OffRamp: verified + OffRamp-->>OffRamp: emit MessageExecuted(messageId) +``` + +## Code Pointers + +### Contracts + +- `contracts/src/ccv/SymbioticCCV.sol` -- CCV verifier implementation (`verifyMessage`, `forwardToVerifier`) +- `contracts/src/ccv/interfaces/` -- CCV interface definitions (`ICrossChainVerifierV1`, etc.) +- `contracts/src/ccv/libraries/` -- CCV encoding and helper libraries +- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement +- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry +- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management + +### Operator (Rust) + +- `operator/src/provider/chainlink_ccv.rs` -- Decodes `CCIPMessageSent` events, builds CCV payloads +- `operator/src/provider/mod.rs` -- `Provider` trait and registration + +### Config Templates + +- `config/templates/oz-monitor/monitors/ccip_message_sent.json` -- Monitor job for `CCIPMessageSent` events + +## Configuration + +Select CCV as the active provider: + +```json +// config/environments/.json +{ + "activeProvider": "chainlink_ccv" +} +``` + +Chain config is shared across providers — chain IDs from `chains.source.chainId` and `chains.destination.chainId` are used as CCIP chain selectors at runtime. + +Address resolution for CCV scripts: + +1. `CCV_*` environment variables (highest priority) +2. `deployments/.json` + +CCV settlement addresses in deployment state: + +- Destination: `destination.chainlinkCcv.settlement` in `deployments/.json` + +Available `CCV_*` override variables: + +| Variable | Description | +|----------|-------------| +| `CCV_SOURCE_ADDRESS` | SymbioticCCV on source chain | +| `CCV_DEST_ADDRESS` | SymbioticCCV on destination chain | +| `CCV_SOURCE_ONRAMP_ADDRESS` | Source OnRamp-compatible contract | +| `CCV_DEST_OFFRAMP_ADDRESS` | Destination OffRamp submit target | + +## Usage + +```bash +# Select chainlink_ccv provider in config/environments/local.json +# "activeProvider": "chainlink_ccv" + +# Start the stack +make start + +# Send a test message +make send MSG="hello" + +# Watch until MessageExecuted on destination +make watch + +# Or run both +make e2e +``` + +`make send` sends through the source mock `OnRamp.sendMessage(...)`, emitting `CCIPMessageSent`. + +`make watch` succeeds only when `MessageExecuted(messageId)` is found on the destination chain (not just relayer submission). + +See [CLI Reference](/symbiotic/cli) for full command options. + +## Deployment Status + +| Environment | Status | +|-------------|--------| +| Local | Supported (Symbiotic-only mock path) | +| Testnet | Not yet | +| Mainnet | Not yet | + +## Common Issues + +- **EpochTooStale revert (0xf5ab0d81)** -- Settlement epoch data is stale. Refresh genesis or tune epoch timing. See [Troubleshooting](/symbiotic/troubleshooting#epochtoostale-revert-0xf5ab0d81). +- **Watch does not reach success** -- CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. See [Troubleshooting](/symbiotic/troubleshooting#watch-does-not-reach-success). +- **Submission fails at estimate-gas** -- Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. See [Troubleshooting](/symbiotic/troubleshooting#submission-fails-at-estimate-gas). diff --git a/content/symbiotic/cli.mdx b/content/symbiotic/cli.mdx new file mode 100644 index 00000000..dc1607f2 --- /dev/null +++ b/content/symbiotic/cli.mdx @@ -0,0 +1,268 @@ +--- +title: CLI & API Reference +--- + +Command-line interface and HTTP API for the Symbiotic multi-provider template. + +## Make Commands + +### `make send` + +Send one test message. Provider-aware based on `activeProvider`. + +```bash +make send +make send MSG="test message" +make send ENV=testnet MSG="hello" +``` + +### `make watch` + +Watch a previously sent message until it lands on the destination chain. + +```bash +make watch +make watch ENV=testnet TIMEOUT=300 +make watch GUID=0x... +make watch TX=0x... +``` + +| Variable | Description | +|----------|-------------| +| `ENV` | Environment (default: local) | +| `TIMEOUT` | Max wait in seconds | +| `GUID` | Watch specific message by GUID | +| `TX` | Watch message by source TX hash | + +### `make e2e` + +Send a message, then watch it to completion. + +```bash +make e2e +make e2e MSG="custom message" +make e2e ENV=testnet MSG="hello" TIMEOUT=180 +``` + +Example output: + +``` +[18:53:21] Operators: waiting to batch +[18:53:28] Operators: collecting BLS signatures +[18:53:30] Operators: signed (quorum reached) +[18:53:32] Relayer: submitted +[18:53:34] Relayer: confirmed (tx: 0x4617...) +[18:53:34] Destination target: verified on-chain (tx: 0x4617...) + +Message verified on destination chain! +``` + +## Direct xtask Commands + +Use the Rust CLI directly: + +```bash +cargo xtask --env local msg send "hello" +cargo xtask --env local msg watch --timeout 120 +cargo xtask --env local msg e2e "hello" --timeout 120 +``` + +### `cargo xtask msg send` + +```bash +cargo xtask --env local msg send "hello" +cargo xtask --env local msg send "hello" --gas 250000 +cargo xtask --env local msg send "hello" --json +``` + +### `cargo xtask msg watch` + +```bash +cargo xtask --env local msg watch +cargo xtask --env local msg watch --id 0x... +cargo xtask --env local msg watch --tx 0x... +cargo xtask --env local msg watch --timeout 300 +``` + +### `cargo xtask msg e2e` + +```bash +cargo xtask --env local msg e2e "hello" +cargo xtask --env local msg e2e "hello" --timeout 300 +cargo xtask --env local msg e2e "hello" --json +``` + +## Message Cache + +After `send`, xtask saves message details to: + +``` +generated//msg-cache.json +``` + +`watch` uses this cache when no explicit `--id` or `--tx` is provided. + +## HTTP API + +Each operator exposes HTTP endpoints on ports 3001-3003. + +### Webhook Endpoints + +#### POST /webhook/events + +Receives provider ingress events from OZ Monitor. + +**Authentication:** HMAC-SHA256 via two headers: +- `X-Signature`: Hex-encoded HMAC-SHA256 of `body + timestamp` +- `X-Timestamp`: Unix timestamp in milliseconds + +The webhook secret must match between operator (`WEBHOOK_SECRET` env var) and OZ Monitor trigger config (`config.secret.value`). + +#### POST /api/v1/webhooks/oz-relayer + +Receives transaction status updates from OZ Relayer. + +**Authentication:** Base64-encoded HMAC-SHA256 of raw JSON body in `X-Signature` header, using `OZ_RELAYER_WEBHOOK_SECRET`. + +### Debug Endpoints + +#### GET /debug/v1/messages + +List messages with processing and submission status. + +```bash +curl -s http://localhost:3001/debug/v1/messages +curl "http://localhost:3001/debug/v1/messages?status=pending&limit=10" +``` + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `status` | (all) | Filter: `pending`, `processing`, `signed` | +| `limit` | 50 | Max messages returned | +| `offset` | 0 | Pagination offset | + +#### GET /debug/v1/messages/:message_id + +Get a specific message by ID. + +#### GET /debug/v1/pending + +List Merkle roots awaiting BLS signatures. + +### Proof Endpoints + +#### POST /api/v1/layerzero/proof + +Retrieve Merkle proofs for processed messages. + +```bash +curl -X POST http://localhost:3001/api/v1/layerzero/proof \ + -H "Content-Type: application/json" \ + -d '{"message_ids": ["0xabc123..."]}' +``` + +Response fields: `root_hash`, `root_proof` (BLS signature), `index`, `leaf`, `siblings`, `original_list`. + +#### POST /api/v1/layerzero/verify + +Verify a Merkle proof is valid (testing only). + +### GET /healthz + +Returns `200 OK` if healthy. + +## Webhook Configuration + +OZ Monitor sends events via HMAC-SHA256 authenticated webhooks. + +### Trigger Template + +Webhook triggers are defined in `config/templates/oz-monitor/triggers/` and copied to `generated//oz-monitor/triggers/` at startup. + +Key settings: + +| Setting | Description | +|---------|-------------| +| `url.value` | Operator endpoint (use Docker service name in compose) | +| `secret.value` | Must match `WEBHOOK_SECRET` in operator `.env` | +| `payload_mode` | Must be `"raw"` | + +### Monitor Jobs + +Provider-specific monitor templates: +- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` +- `config/templates/oz-monitor/monitors/ccip_message_sent.json` + +### Webhook Payload Format + +```json +{ + "EVM": { + "logs": [{ "address": "0x...", "topics": ["..."], "data": "0x..." }], + "matched_on_args": { + "events": [{ + "signature": "JobAssigned(address,bytes,uint256,address)", + "args": [{ "name": "dvn", "kind": "address", "value": "0x..." }] + }] + }, + "monitor": { "name": "LayerZero JobAssigned" }, + "network_slug": "anvil-source", + "transaction": { "hash": "0x...", "blockNumber": 123 } + } +} +``` + +For CCV, the event signature is `CCIPMessageSent(...)` with different args. + +## Retry Configuration + +### Symbiotic Relay (Linear Backoff) + +For gRPC calls to BLS signing sidecars. + +``` +backoff = retry_backoff x (attempt + 1) +``` + +### OZ Relayer (Exponential Backoff with Jitter) + +For HTTP calls to the transaction relayer. + +``` +base = retry_backoff x 2^attempt +jitter = random(0, base x 0.25) +backoff = min(base + jitter, 60s) +``` + +### Retry Settings + +| Setting | Description | Default | +|---------|-------------|---------| +| `max_retries` | Maximum retry attempts (0 = no retries) | 3 | +| `retry_backoff` | Base backoff duration | 1s | +| `timeout` | Request timeout (OZ Relayer only) | 30s | + +### Retryable vs Non-Retryable Errors + +**Retried:** HTTP 429, HTTP 500-504, network errors (connection refused, timeout, DNS failure). + +**Not retried:** HTTP 4xx (except 429), domain errors (chain not configured, transaction not found). + +### Tuning Examples + +```json +// Low-latency (devnet/testnet) +{ "oz_relayer": { "max_retries": 5, "retry_backoff": "100ms" } } + +// Production +{ "oz_relayer": { "max_retries": 3, "retry_backoff": "1s" } } + +// High-volume +{ "oz_relayer": { "max_retries": 5, "retry_backoff": "2s" } } +``` + +| Config | Symbiotic Relay total | OZ Relayer total | +|--------|----------------------|------------------| +| 1s / 3 retries | ~6s | ~9s | +| 1s / 5 retries | ~15s | ~39s | +| 2s / 3 retries | ~12s | ~18s | diff --git a/content/symbiotic/deployment.mdx b/content/symbiotic/deployment.mdx new file mode 100644 index 00000000..f5958481 --- /dev/null +++ b/content/symbiotic/deployment.mdx @@ -0,0 +1,168 @@ +--- +title: Deployment +--- + +Deploying and operating the stack beyond local development. + + + +Testnet deployment currently supports the `layerzero` provider only. CCV is local-only for now. + + + +## Testnet (LayerZero) + +- Source: Base Sepolia (`84532`) +- Destination: Sepolia (`11155111`) +- Provider: `layerzero` + +### Runtime Model + +```bash +make validate ENV=testnet +make deploy ENV=testnet +make refresh-genesis ENV=testnet # when validation says genesis is stale +make run-operators ENV=testnet +make e2e ENV=testnet +``` + +| Command | What it does | +|---------|-------------| +| `validate` | Read-only checks: config, chain reachability, deployment state, operator state, relayer signer safety | +| `deploy` | Deploy managed contracts, update `deployments/testnet.json` and `generated/testnet/` | +| `refresh-genesis` | Refresh settlement genesis without redeploying contracts | +| `run-operators` | Start non-local operator-side services | + +### Environment Inputs + +Config lives in `config/environments/testnet.json`: +- Chain IDs and EIDs +- Default RPC URLs +- LayerZero predeploys +- Symbiotic Core predeploys +- Relay timing + +xtask resolves RPC URLs from the environment JSON first. `SOURCE_RPC_URL` / `DEST_RPC_URL` in `.env` are fallback overrides only. + +Required `.env` values: + +```bash +PRIVATE_KEY=0x +KEYSTORE_PASSPHRASE= +OPERATOR_1_PRIVATE_KEY=0x +OPERATOR_2_PRIVATE_KEY=0x +OPERATOR_3_PRIVATE_KEY=0x +``` + +Optional relayer bootstrap inputs: + +```bash +RELAYER_1_PRIVATE_KEY=0x +RELAYER_2_PRIVATE_KEY=0x +RELAYER_3_PRIVATE_KEY=0x +``` + +Relayer private keys are setup-time inputs only. The steady-state runtime source is the OZ relayer keystore files under `config/oz-relayer/keys/`. + +### Workflow + +#### 1. Generate keys + +```bash +make setup +``` + + + +For public testnets: do not use known local/dev keys. Ensure deployer, operators, and relayer signers all have testnet ETH. + + + +#### 2. Validate first + +```bash +make validate ENV=testnet +``` + +Catches: missing keys, underfunded accounts, stale genesis, relayer signer issues. + +#### 3. Deploy managed contracts + +```bash +make deploy ENV=testnet +``` + +Updates `deployments/testnet.json` and `generated/testnet/`. + +#### 4. Refresh genesis when needed + +```bash +make refresh-genesis ENV=testnet +``` + +Use this instead of redeploying when contracts are already in place and only the settlement header is stale. + +#### 5. Start operator services + +```bash +make run-operators ENV=testnet +``` + +Starts from `docker-compose.yml` (no `docker-compose.local.yml` overlay): +- Operators, OZ Monitor, OZ Relayer, Symbiotic relay sidecars + +#### 6. Send and verify + +```bash +make e2e ENV=testnet MSG="hello" +``` + +### How Testnet Differs From Local + +| Area | Local | Testnet | +|------|-------|---------| +| Entrypoint | `make start` | `make deploy` + `make run-operators` | +| Chains | Local Anvil | Base Sepolia + Sepolia | +| LayerZero endpoints | Local mocks | Predeployed | +| Symbiotic Core | Deployed fresh | Predeployed on destination | +| Genesis refresh | Folded into startup | Explicit `make refresh-genesis` when stale | +| Compose files | `docker-compose.yml` + `docker-compose.local.yml` | `docker-compose.yml` only | +| Operator registration | Auto-impersonate | Separate step (real chains) | + +### Testnet Architecture + +```text +Base Sepolia (84532) Sepolia (11155111) +-------------------- ------------------- +LZ V2 Endpoint (pre-deployed) LZ V2 Endpoint (pre-deployed) +DVN.assignJob() DVN.submitProof() -> Settlement +TestOApp.send() TestOApp.lzReceive() + Driver, KeyRegistry, VotingPowers + + OZ Monitor -> Operators -> Symbiotic Relays -> OZ Relayer + (local Docker containers) +``` + +### Testnet Troubleshooting + +**`validation failed: genesis stale`** -- Run `make refresh-genesis ENV=testnet`. + +**`insufficient funds for gas`** -- Fund the deployer address from `PRIVATE_KEY`. + +**Sidecars fail with RPC rate limits** -- Three sidecars syncing at once can overwhelm weak testnet RPC plans. Use higher-throughput RPCs or temporarily reduce sidecar count. + +**Keys look drained immediately** -- Do not use known local/dev keys on public testnets. Regenerate with `make setup`. + +**Fresh relay deploys on shared testnet** -- Most fragile path. Prefer reusing existing environment and `make refresh-genesis` for stale-genesis repair. + +## Mainnet + +Not yet supported. Key differences from testnet will include: +- Hardware security for BLS keys +- Distributed operators (not co-located containers) +- OZ-hosted services +- Configurable quorum thresholds + +## Address Management + +All deployment addresses are canonical in `deployments/.json`. This file is updated by `make deploy` and read by all runtime components. diff --git a/content/symbiotic/index.mdx b/content/symbiotic/index.mdx new file mode 100644 index 00000000..706845bc --- /dev/null +++ b/content/symbiotic/index.mdx @@ -0,0 +1,26 @@ +--- +title: Symbiotic Templates +--- + +## For Operators + +Running, configuring, and monitoring the stack. + +1. [Setup](/symbiotic/setup) -- Config structure, environment setup, running locally +2. [Deployment](/symbiotic/deployment) -- Testnet and mainnet deployment +3. Choose your provider: + - [LayerZero](/symbiotic/layerzero) -- DVN for LayerZero V2 + - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Cross-Chain Verifier for CCIP +4. [CLI & API Reference](/symbiotic/cli) -- Commands, HTTP endpoints, webhook config +5. [Troubleshooting](/symbiotic/troubleshooting) -- Common issues and debugging + +## For Integrators + +Understanding the system and adding new providers. + +1. [Architecture](/symbiotic/architecture) -- Provider model, shared infra, Merkle batching, BLS signing +2. Choose your provider: + - [LayerZero](/symbiotic/layerzero) -- Message flow, contracts, code pointers + - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Message flow, contracts, code pointers +3. [Architecture: Adding a New Provider](/symbiotic/architecture#adding-a-new-provider) -- Provider trait, registration, templates +4. [Security](/symbiotic/security) -- Trust model, access control, invariants diff --git a/content/symbiotic/layerzero.mdx b/content/symbiotic/layerzero.mdx new file mode 100644 index 00000000..91e4d569 --- /dev/null +++ b/content/symbiotic/layerzero.mdx @@ -0,0 +1,128 @@ +--- +title: LayerZero +--- + +Symbiotic-secured DVN (Decentralized Verifier Network) for LayerZero V2 cross-chain messaging. + +## Overview + +The LayerZero provider implements a DVN that uses Symbiotic shared security to verify cross-chain messages. When a message is sent through LayerZero's `SendUln302`, the DVN contract emits a `JobAssigned` event. Operators batch these jobs into Merkle trees, collect BLS signatures through Symbiotic relay sidecars, and submit the signed proof to the destination DVN contract. The destination DVN verifies the BLS quorum via the Settlement contract and forwards verification to LayerZero's `ReceiveUln302`. + +## Message Flow + +```mermaid +sequenceDiagram + participant App as User App + participant SendUln as SendUln302 (Source) + participant DVN_S as DVN.assignJob (Source) + participant Monitor as OZ Monitor + participant Operators as Operators (x3) + participant Relay as Symbiotic Relay (BLS) + participant Relayer as OZ Relayer + participant DVN_D as DVN.submitProof (Dest) + participant Settlement as Settlement (BLS verify) + participant RecvUln as ReceiveUln302 (Dest) + + App->>SendUln: send message + SendUln->>DVN_S: assignJob() + DVN_S-->>Monitor: JobAssigned event + Monitor->>Operators: HMAC webhook + Operators->>Operators: batch into Merkle tree + Operators->>Relay: sign Merkle root (BLS) + Relay-->>Operators: aggregated signature + Operators->>Relayer: submitProof calldata + Relayer->>DVN_D: submitProof(root, proof, signatures) + DVN_D->>Settlement: verify BLS quorum + Settlement-->>DVN_D: quorum valid + DVN_D->>RecvUln: verify() +``` + +## Code Pointers + +### Contracts + +- `contracts/src/SymbioticLayerZeroDVN.sol` -- DVN contract handling `assignJob` (source) and `submitProof` (destination) +- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement +- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry +- `contracts/src/symbiotic/VotingPowers.sol` -- Operator voting power tracking +- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management +- `contracts/src/examples/TestOApp.sol` -- Test application for sending/receiving messages + +### Operator (Rust) + +- `operator/src/provider/layerzero.rs` -- Decodes `JobAssigned` events, stores messages +- `operator/src/provider/mod.rs` -- `Provider` trait and registration +- `operator/src/crypto/mod.rs` -- Merkle tree construction, DVN leaf hashing +- `operator/src/signer/mod.rs` -- Batches messages, requests BLS signatures +- `operator/src/relay_submitter/mod.rs` -- Submits signed proofs via OZ Relayer + +### Config Templates + +- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` -- Monitor job for `JobAssigned` events +- `config/templates/oz-monitor/triggers/webhook_layerzero.json` -- Webhook trigger template + +## Configuration + +Select LayerZero as the active provider: + +```json +// config/environments/.json +{ + "activeProvider": "layerzero" +} +``` + +Chain config is shared across providers and lives at the top level: + +| Field | Description | +|-------|-------------| +| `chains.source.chainId` | Source chain ID | +| `chains.destination.chainId` | Destination chain ID | +| `chains.source.eid` | LayerZero endpoint ID for source | +| `chains.destination.eid` | LayerZero endpoint ID for destination | + +LayerZero predeploys (testnet/mainnet) go in `chains..predeploys.layerzero`: + +```json +{ + "predeploys": { + "layerzero": { + "endpoint": "0x6EDCE65403992e310A62460808c4b910D972f10f", + "sendUln302": "0xC1868e054425D378095A003EcbA3823a5D0135C9" + } + } +} +``` + +`make deploy` and `make start` use these values to generate runtime configs (`destination_chains`, `chain_relayers`, `eid_to_chain_id`) under `generated//`. Validation fails if chain IDs/EIDs drift from the generated deployment state. + +## Usage + +```bash +# Select layerzero provider in config/environments/local.json +# "activeProvider": "layerzero" + +# Start the stack +make start + +# Send a test message +make send MSG="hello" + +# Watch until destination verification +make watch + +# Or run both in one shot +make e2e +``` + +`make send` sends through `TestOApp.send(...)` which calls `SendUln302`, triggering `DVN.assignJob()`. + +`make watch` succeeds when destination target verification is observed on-chain. + +See [CLI Reference](/symbiotic/cli) for full command options. + +## Common Issues + +- **Message stuck at "Processing"** -- BLS signatures not aggregating. Check sidecar health and operator key registration. See [Troubleshooting](/symbiotic/troubleshooting#bls-signatures-not-aggregating). +- **Quorum not reached** -- All 3 operators must be running and receiving the same events. See [Troubleshooting](/symbiotic/troubleshooting#quorum-not-reached). +- **submitProof reverts** -- Check that the OZ Relayer address is authorized as a submitter on the DVN contract, and that Settlement has correct operator keys. See [Troubleshooting](/symbiotic/troubleshooting#layerzero-issues). diff --git a/content/symbiotic/security.mdx b/content/symbiotic/security.mdx new file mode 100644 index 00000000..7df1803e --- /dev/null +++ b/content/symbiotic/security.mdx @@ -0,0 +1,113 @@ +--- +title: Security +--- + +Security architecture and trust assumptions for the Symbiotic multi-provider template. + +## Shared Trust Model + +| Entity | Trust Level | Notes | +|--------|-------------|-------| +| **Settlement** | Trusted | Symbiotic contract for BLS signature verification | +| **Authorized Submitters** | Semi-trusted | Whitelisted addresses that submit proofs; cannot forge signatures but can grief (spam invalid proofs) | +| **Owner** | Trusted | Admin with pause/unpause, submitter management | +| **External users** | Untrusted | Cannot call privileged functions directly | + +### Symbiotic Security Layer + +- Operators stake and register BLS public keys via `KeyRegistry` +- Settlement contract verifies BLS quorum before accepting proofs +- Slashing handled by Symbiotic core contracts (production) + +### Webhook Authentication + +Webhooks between OZ Monitor and operators use HMAC-SHA256: +1. Monitor computes `HMAC-SHA256(secret, body + timestamp)` +2. Signature sent in `X-Signature` header +3. Timestamp (ms since epoch) sent in `X-Timestamp` header +4. Operator rejects invalid/missing signatures or expired timestamps (HTTP 401) + +Secrets must be at least 32 characters. See [CLI Reference](/symbiotic/cli#webhook-configuration) for config details. + +## LayerZero DVN Security + +Trust assumptions specific to the `SymbioticLayerZeroDVN` contract. + +| Entity | Trust Level | Notes | +|--------|-------------|-------| +| **SendUln302** | Trusted | LayerZero's send library; only caller for `assignJob` | + +### Access Control + +#### Source Chain + +| Function | Caller | Purpose | +|----------|--------|---------| +| `assignJob` | SendUln302 only | Register verification job, emit event | +| `getFee` | Anyone | Query verification fee (view) | + +#### Destination Chain + +| Function | Caller | Purpose | +|----------|--------|---------| +| `submitProof` | Authorized submitters | Submit signed Merkle proof for verification | + +#### Admin + +| Function | Caller | Purpose | +|----------|--------|---------| +| `addSubmitter` / `removeSubmitter` | Owner | Manage submitter whitelist | +| `setBaseFee` | Owner | Update verification fee | +| `pause` / `unpause` | Owner | Emergency controls | +| `withdraw` | Owner | Recover ETH (force-sent or accidental) | +| `transferOwnership` | Owner | Transfer admin rights | + +### Invariants + +1. **Leaf monotonicity**: `verifiedLeaves[leaf]` transitions `false -> true` only, never back +2. **Root monotonicity**: `verifiedRoots[root]` transitions `false -> true` only, never back +3. **Signature requirement**: Uncached roots require valid BLS quorum from Settlement +4. **Packet header integrity**: Verified packets have exactly 81 bytes and correct `dstEid` +5. **No ETH custody**: Contract does not collect fees; `assignJob` rejects `msg.value > 0` + +### Deployment Modes + +| Mode | sendUln | receiveUln | settlement | Use case | +|------|---------|------------|------------|----------| +| Source only | Set | Zero | Zero | Emit `JobAssigned` events | +| Destination only | Zero | Set | Set | Verify proofs, call ReceiveUln | +| Bidirectional | Set | Set | Set | Both functions on same chain | + +### What the DVN Does NOT Do + +- **Fee custody**: Fees handled by LayerZero's fee accounting +- **Signature generation**: BLS signing happens off-chain via Symbiotic Relay +- **Slashing**: Handled by Symbiotic core contracts + +## Chainlink CCV Security + +Trust assumptions specific to the `SymbioticCCV` contract. + +### Access Control + +| Function | Caller | Purpose | +|----------|--------|---------| +| `forwardToVerifier` | OnRamp | Source-chain hook for CCV registration | +| `verifyMessage` | OffRamp | Destination verification hook | +| `getFee` | Anyone | Quote verification fee (view) | + +### Invariants + +CCV verification requires: +1. Valid BLS quorum signature from Settlement +2. Correct message ID derivation +3. Epoch freshness (reverts with `EpochTooStale` if settlement data is stale) + +## External Dependencies + +| Dependency | Version | Purpose | +|------------|---------|---------| +| `@openzeppelin/contracts` | 5.x | MerkleProof verification | +| `@symbioticfi/relay-contracts` | - | Settlement base contracts | +| LayerZero V2 | - | ILayerZeroDVN interface | +| Chainlink CCIP | - | CCV interfaces (ICrossChainVerifierV1) | diff --git a/content/symbiotic/setup.mdx b/content/symbiotic/setup.mdx new file mode 100644 index 00000000..bbbba4ae --- /dev/null +++ b/content/symbiotic/setup.mdx @@ -0,0 +1,183 @@ +--- +title: Setup +--- + +Getting the stack running locally. + +## Prerequisites + +- Docker and Docker Compose v2+ +- [Foundry](https://book.getfoundry.sh/getting-started/installation) (`forge`, `cast`, `anvil`) +- [Rust/Cargo](https://rustup.rs/) (for `make dev-operator`) +- `jq` + +## Config Structure + +``` +config/ +├── environments/ # Per-network config (local.json, testnet.json) +├── templates/ # Service config templates (oz-monitor, oz-relayer) +├── oz-monitor/ # Static monitor config +└── oz-relayer/ # Static relayer config and keystores + +deployments/ +└── .json # Canonical deployment addresses + +generated/ +└── / # Generated runtime config and message cache +``` + +**How it works:** +1. `make deploy` deploys contracts and updates `deployments/.json` +2. `make deploy` and `make start` generate provider-specific runtime config under `generated//` +3. Docker containers mount from `generated//` + +**To customize configs:** Edit templates in `config/templates/`, then rerun `make deploy` or `make start`. + +## Environment Setup + +Bootstrap local `.env` and operator keys: + +```bash +make setup +``` + +This generates `.env` from `.env.example` and creates BLS keys and relayer keystores. + + + +`make start` also auto-bootstraps `.env` and keystores if missing, so `make setup` is only needed for explicit regeneration. + + + +### Environment Variables + +| Variable | Description | +|----------|-------------| +| `PRIVATE_KEY` | Deployer key (default: Anvil account 0) | +| `LOG_LEVEL` | Logging level (debug, info, warn, error) | +| `WEBHOOK_SECRET` | HMAC secret for webhook auth (min 32 chars) | +| `OZ_RELAYER_WEBHOOK_SECRET` | Secret for OZ Relayer webhook auth (min 32 chars) | +| `OZ_RELAYER_API_KEY` | **Required.** Relayer API authentication | +| `SIDECAR_*_SECRET_KEYS` | BLS keys per operator (generated) | + +Generate secrets with: + +```bash +openssl rand -hex 32 +``` + + + +The operator will fail to start if `WEBHOOK_SECRET`, `OZ_RELAYER_WEBHOOK_SECRET`, or `OZ_RELAYER_API_KEY` are missing. Secrets must be at least 32 characters. + + + +## Provider Selection + +Provider is set in `config/environments/.json`: + +```json +{ + "activeProvider": "layerzero" +} +``` + +All `make` commands (`start`, `send`, `watch`, `e2e`) are provider-aware based on this field. + +See provider-specific config in [LayerZero](/symbiotic/layerzero#configuration) or [Chainlink CCV](/symbiotic/chainlink-ccv#configuration). + +## Running Locally + +```bash +# Start the full stack (auto-bootstrap + deploy + start services) +make start + +# Check service health +make status + +# Send a test message and watch it complete +make e2e +``` + +### Common Commands + +``` +make start Start the full local stack +make deploy Deploy contracts and generate service config +make stop Stop all containers (preserve state) +make clean Full reset (stop + remove volumes + markers) + +make restart-operators Rebuild and restart all 3 operators +make restart-monitor Restart oz-monitor (config reload) +make restart-relayer Restart oz-relayer +make restart-relays Restart symbiotic-relay-1/2/3 + +make dev-operator Run operator-1 locally (cargo run) +make rebuild-operators Docker rebuild + restart all operators +make shell Interactive shell with addresses loaded + +make test Run unit tests (forge + cargo) +make test-contracts Run contract tests only + +make logs-operators Follow all 3 operator logs +make logs-operator-N Follow operator-N logs (N=1,2,3) +make logs-monitor Follow oz-monitor logs +make logs-relayer Follow oz-relayer logs +make logs-relays Follow symbiotic-relay-1/2/3 logs + +make status Show running containers and health +make help Show all available commands +``` + +### Local Operator Development + +Run operator-1 outside Docker for fast iteration: + +```bash +# Start the full stack first +make start + +# Run operator-1 locally (replaces the Docker container) +make dev-operator +``` + +This runs `cargo run` with the generated config and `RUST_LOG=debug`. The local operator connects to the same Docker services and receives the same webhooks. + +## Operator Configuration + +Runtime configs are generated at `generated//operator-{n}/config.json`. + +Key settings: + +| Setting | Default | Description | +|---------|---------|-------------| +| `signer.event_poll_interval` | 15s | How often to check for new pending messages | +| `signer.sign_job_interval` | 1s | How often to retry pending Merkle roots | +| `signer.sign_worker_count` | 5 | Concurrent signing workers | +| `signer.min_batch_size` | 1 | Minimum messages before creating a tree | +| `oz_relayer.poll_interval` | 5s | How often to check for signed trees to submit | +| `oz_relayer.status_poll_interval` | 30s | How often to poll OZ Relayer for tx status | +| `symbiotic_relay.key_tag` | 15 | BLS key identifier in the sidecar | + +## Contract Addresses + +After deployment, canonical addresses are in `deployments/.json`. + +For manual testing, `make shell` opens an interactive shell with `.env` sourced and `ENV_CONFIG` / `DEPLOYMENTS_FILE` exported: + +```bash +make shell +# Then use jq to extract addresses: +jq '.source.layerzero.dvn' $DEPLOYMENTS_FILE +``` + +## Service Ports + +| Service | Port | Purpose | +|---------|------|---------| +| anvil (source) | 8545 | Source chain RPC | +| anvil (dest) | 8546 | Destination chain RPC | +| operator-1/2/3 | 3001-3003 | Operator debug APIs | +| symbiotic-relay-1/2/3 | 8081-8083 | BLS sidecars | +| oz-relayer | 8080 | Transaction relayer | diff --git a/content/symbiotic/troubleshooting.mdx b/content/symbiotic/troubleshooting.mdx new file mode 100644 index 00000000..f3e8f131 --- /dev/null +++ b/content/symbiotic/troubleshooting.mdx @@ -0,0 +1,187 @@ +--- +title: Troubleshooting +--- + +Common issues and solutions for the Symbiotic multi-provider template. + +## Checking Message Status + +```bash +# List all messages +curl http://localhost:3001/debug/v1/messages + +# Filter by status +curl "http://localhost:3001/debug/v1/messages?status=pending" + +# Get specific message +curl http://localhost:3001/debug/v1/messages/0xabc123... + +# Check pending Merkle roots +curl http://localhost:3001/debug/v1/pending +``` + +## Shared Issues + +### Webhook Not Received + +1. Check operator logs: + ```bash + make logs-operators | grep webhook + ``` +2. Verify connectivity (operators must be reachable from OZ Monitor): + ```bash + docker compose exec oz-monitor curl -s http://operator-1:3000/healthz + ``` +3. Confirm trigger is linked in the active monitor config under `generated//oz-monitor/monitors/` and rerun `make deploy` or `make start` if needed. + +### Authentication Failures (401) + +1. Verify secrets match between OZ Monitor config and operator `.env` +2. Ensure secret is at least 32 characters +3. Check for trailing whitespace in config files + +### Payload Parsing Errors + +1. Ensure `payload_mode: "raw"` is set in trigger config +2. Check operator logs for deserialization errors + +### BLS Signatures Not Aggregating + +1. Check sidecar status: + ```bash + docker compose ps symbiotic-relay-1 + docker compose logs symbiotic-relay-1 + ``` +2. Verify BLS keys in `.env` +3. Verify operator keys are registered in the Settlement contract + +### Quorum Not Reached + +1. All required operators must be running +2. Check operators are receiving the same events +3. Verify keys are registered in Settlement + +### Relayer Request Failed After All Retries + +1. Check OZ Relayer: + ```bash + curl -H "Authorization: Bearer $OZ_RELAYER_API_KEY" http://localhost:8080/api/v1/health + ``` +2. Check relayer logs: + ```bash + make logs-relayer + ``` +3. For 429 errors: increase `retry_backoff` or reduce submission rate + +### Service Won't Start + +1. Check missing env vars: + ```bash + docker compose config + ``` +2. Verify secrets are at least 32 characters +3. Check port conflicts: + ```bash + lsof -i :3001 # operator + lsof -i :8080 # relayer + ``` + +### Anvil Not Responding + +```bash +docker compose restart anvil anvil-settlement +docker compose logs anvil +``` + +### Contracts Not Found + +```bash +cat deployments/local.json +# Re-deploy if needed: +make clean && make start +``` + +### First-Run Genesis Waits + +On fresh devnet, `make start` may wait while voting power is captured. + +Symptoms: logs show `genesis not ready: totalVotingPower 0 < quorumThreshold 1`. + +This is expected on clean boot. If stuck, reset: `make clean && make start`. + +## LayerZero Issues + +### Message Stuck at "Processing" + +BLS signing not completing. Check sidecar health and logs: + +```bash +docker compose ps symbiotic-relay-1 +docker compose logs symbiotic-relay-1 +``` + +### submitProof Reverts + +Check Settlement contract state (operator keys, quorum threshold). Inspect: + +```bash +make logs-relayer | grep -i "revert\|error" +``` + +## Chainlink CCV Issues + +### EpochTooStale Revert (0xf5ab0d81) + +Settlement epoch/timestamp data is stale. Relayer fails at gas estimation. + +```bash +make logs-relayer | grep -E "estimate_gas|custom error|0xf5ab0d81" +``` + +Fix: refresh genesis or tune epoch timing parameters. + +### Watch Does Not Reach Success + +CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. + +```bash +# Verify provider selection +jq -r '.activeProvider' config/environments/local.json + +# Check message lifecycle +make watch + +# If Failed, inspect relayer +make logs-relayer +``` + +### Submission Fails at estimate-gas + +```bash +make logs-relayer | grep -E "estimate_gas|custom error" +``` + +Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. + +## Log Analysis + +### Useful Commands + +```bash +make logs-operators # All operator logs +make logs-operators 2>&1 | grep -i error # Errors only +make logs-operators | grep -i "retrying" # Retry activity +make logs-operators | grep "retries exhausted" +make logs-monitor # OZ Monitor +make logs-relayer # OZ Relayer +``` + +### Common Log Patterns + +| Pattern | Meaning | +|---------|---------| +| `webhook received` | Event received from OZ Monitor | +| `message batched` | Message added to Merkle tree | +| `signatures aggregated` | BLS quorum reached | +| `proof submitted` | Sent to OZ Relayer | +| `tx confirmed` | On-chain confirmation | diff --git a/src/navigation/ethereum-evm.json b/src/navigation/ethereum-evm.json index 8ce1463b..5644007d 100644 --- a/src/navigation/ethereum-evm.json +++ b/src/navigation/ethereum-evm.json @@ -797,6 +797,57 @@ "type": "separator", "name": "Open Source Tools" }, + { + "type": "folder", + "name": "Symbiotic Templates", + "index": { + "type": "page", + "name": "Overview", + "url": "/symbiotic" + }, + "children": [ + { + "type": "page", + "name": "Setup", + "url": "/symbiotic/setup" + }, + { + "type": "page", + "name": "Architecture", + "url": "/symbiotic/architecture" + }, + { + "type": "page", + "name": "LayerZero", + "url": "/symbiotic/layerzero" + }, + { + "type": "page", + "name": "Chainlink CCV", + "url": "/symbiotic/chainlink-ccv" + }, + { + "type": "page", + "name": "CLI & API Reference", + "url": "/symbiotic/cli" + }, + { + "type": "page", + "name": "Deployment", + "url": "/symbiotic/deployment" + }, + { + "type": "page", + "name": "Security", + "url": "/symbiotic/security" + }, + { + "type": "page", + "name": "Troubleshooting", + "url": "/symbiotic/troubleshooting" + } + ] + }, { "type": "folder", "name": "Relayer",