-
Notifications
You must be signed in to change notification settings - Fork 198
Docs: Node comparison report #7346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
5b00dd2
c6b4474
9364967
fa62a21
0f80b99
77769dd
2aa724f
3da2abb
68b060e
37d8205
12dc461
937c968
4003d95
cc54a72
e583ad3
e2de7da
5a4ae0f
9295fbe
8734aee
bfae341
e153bcf
1d2376d
9ef09ca
4b2890b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "label": "Node Comparison", | ||
| "position": 2 | ||
| } |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,40 @@ | ||||||
| --- | ||||||
| sidebar_position: 1 | ||||||
| title: Filecoin Node Comparison | ||||||
| --- | ||||||
|
|
||||||
| # Filecoin Node Comparison | ||||||
|
|
||||||
| There are three full-node implementations of the Filecoin protocol. | ||||||
|
|
||||||
| - **[Forest](https://github.com/ChainSafe/forest)** is written in Rust and maintained by [ChainSafe Systems](https://chainsafe.io). It focuses on chain validation, a high-performance RPC API, and snapshot generation, all with low hardware requirements. | ||||||
| - **[Lotus](https://github.com/filecoin-project/lotus)** is the reference implementation, written in Go. It provides the complete Filecoin feature set, including the storage-provider stack, and is typically where new protocol features land first. | ||||||
| - **[Venus](https://github.com/filecoin-project/venus)** is a modular Go implementation. Rather than a single daemon, it is a set of independently deployable components (`venus`, `damocles`, `sophon-miner`, `venus-wallet`, `sophon-messager`, `sophon-auth`, `sophon-gateway`) that together pioneered Filecoin's distributed storage pool model and offer a largely Lotus-compatible API. It aims to help small and medium-sized storage providers join the Filecoin network with a lower barrier to entry. | ||||||
|
|
||||||
| ## Feature comparison | ||||||
|
|
||||||
| | Capability | Forest | Lotus | | ||||||
| | ----------------------------------------------------------------- | ------------- | ----- | | ||||||
| | Chain synchronization and validation | Yes | Yes | | ||||||
| | Filecoin JSON-RPC API | Yes | Yes | | ||||||
| | Ethereum-compatible RPC (`eth_*`) | Yes | Yes | | ||||||
| | Snapshot export | Yes | Yes | | ||||||
| | Built-in and standalone wallet (`forest-wallet` / `lotus-wallet`) | Yes | Yes | | ||||||
| | Bootstrap node | Yes | Yes | | ||||||
| | F3 (Fast Finality) participation | Yes | Yes | | ||||||
| | Built-in health-check endpoints | Yes | Yes | | ||||||
| | Storage provider / sealing (`lotus-miner`, `lotus-worker`) | No | Yes | | ||||||
| | Block production / mining | No (untested) | Yes | | ||||||
| | Single binary for all networks (`--chain`) | Yes | No | | ||||||
| | Auto-download snapshot on startup | Yes | No | | ||||||
| | Lite / diff / archival snapshot generation | Yes | No | | ||||||
| | Offline RPC server from snapshot (`forest-tool api serve`) | Yes | No | | ||||||
| | Stateless node mode (`--stateless`) | Yes | No | | ||||||
| | Ethereum `trace_call` and `debug_traceTransaction` | Yes | No | | ||||||
| | Written in Rust 🦀 | Yes | No | | ||||||
|
|
||||||
| Forest and Lotus both expose a compatible Filecoin JSON-RPC API (requests and responses use the same JSON format). Forest supports three API versions: `/rpc/v0` (deprecated; legacy Lotus-compatible methods), `/rpc/v1` (stable and recommended for production), and `/rpc/v2` (experimental; still being rolled out). See the [JSON-RPC overview](../../reference/json-rpc/overview.md), [methods reference](../../reference/json-rpc/methods.mdx), and [API conformance report](../api_conformance/report_2026-07-01.md) for supported methods and Lotus API parity. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Let's assume the reader knows what |
||||||
|
|
||||||
| ## Performance | ||||||
|
|
||||||
| For a comparable RPC workload, Forest serves requests with lower latency while using less CPU and memory than Lotus, and exports snapshots significantly faster while requiring fewer hardware resources. See the [RPC Performance Comparison](./rpc_comparison.md) and [Snapshot Generation Comparison](./snapshot_comparison.md) for the full figures and methodology. | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| --- | ||
| sidebar_position: 2 | ||
| title: RPC Performance Comparison | ||
| --- | ||
|
|
||
| # RPC Performance: Forest vs Lotus | ||
|
|
||
| The following measurements were taken while a Forest node and a Lotus node handled a similar stream of real RPC provider traffic during the same time window. Because both nodes served an almost identical request stream, the differences below are attributable to the nodes themselves rather than to uneven load. The "Comparable load" section documents the checks used to support that assumption. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's mention when exactly they were taken. It's not that important but adds some credibility to those charts. |
||
|
|
||
| ## Environment | ||
|
|
||
| Node builds and hardware used for these measurements: | ||
|
|
||
| | Node | Version | Commit | vCPUs | RAM | | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. let's squash version and commit, no need for an extra column |
||
| | ------ | ------------ | ------------- | ----- | ----- | | ||
| | Forest | `0.34.1` | `git.0ba362e` | 12 | 64 GB | | ||
| | Lotus | `1.36.1-rc1` | `git.45eade8` | 12 | 80 GB | | ||
|
|
||
| ## RPC latency | ||
|
|
||
| Average wall time per HTTP request was approximately 10-15 ms for Forest and 30-50 ms for Lotus over most of the measurement window. Both traces rose toward the end as the load increased, with Forest reaching about 30 ms and Lotus about 48 ms. | ||
|
|
||
|  | ||
|
|
||
| ### Per-flow latency | ||
|
|
||
| At the median (P50), the two nodes were close: approximately 7 ms for Forest and 8 ms for Lotus. The difference was larger in the tail. At P95, Forest stayed near 50 ms and rose to roughly 115 ms under load, while Lotus stayed around 70-85 ms and reached roughly 185 ms. At P99, Forest held around 200 ms and rose to about 350 ms, while Lotus ranged from approximately 240 ms to a peak near 600 ms. | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p50-per-flow-latency.png" | ||
| alt="P50 per-flow latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p95-per-flow-latency.png" | ||
| alt="P95 per-flow latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p99-per-flow-latency.png" | ||
| alt="P99 per-flow latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| ### Amortized latency | ||
|
|
||
| The amortized view accounts for batched requests. It shows the same pattern as the per-flow view: comparable values at P50 and a larger difference at P95, P99, and on average. | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p50-amortized-latency.png" | ||
| alt="P50 amortized latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p95-amortized-latency.png" | ||
| alt="P95 amortized latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/rpc_comparison/p99-amortized-latency.png" | ||
| alt="P99 amortized latency" | ||
| style={{ maxWidth: '680px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
|  | ||
|
|
||
| ### Comparable load | ||
|
|
||
| The request rates reaching each node were similar, supporting a direct comparison between the two nodes. HTTP request counts averaged 45.2 `req/s` for Forest and 45.0 `req/s` for Lotus, with batch-aware counts at 20.4 and 20.3 `req/s`, respectively. Both nodes, therefore, received a comparable volume of traffic. | ||
|
|
||
|  | ||
|
|
||
|  | ||
|
|
||
| Concurrency followed from the latency difference: Forest kept fewer requests in flight at any instant (mean 0.43, peak 4) than Lotus (mean 1.45, peak 8). | ||
|
|
||
|  | ||
|
|
||
| ## Resource usage | ||
|
|
||
| While serving this workload, Forest used approximately 10-15% CPU, and Lotus used approximately 45-55%, briefly spiking to about 80% during the higher-load period. Forest's CPU trace was also less variable. This usage suggests Forest would likely run comfortably on 2-4 `vCPUs`. | ||
|
|
||
|  | ||
|
|
||
| Memory usage was approximately 8-10% for Forest and approximately 25-30% for Lotus, with Lotus briefly reaching about 40% during the higher-load period. | ||
|
|
||
|  | ||
|
|
||
| :::note | ||
| The absolute memory figures are higher than they appear here because of OS page caches, but the relative proportions between the two nodes still hold. | ||
|
sudo-shashank marked this conversation as resolved.
|
||
| ::: | ||
|
|
||
| ## Related | ||
|
|
||
| - [Snapshot Generation Comparison](./snapshot_comparison.md) | ||
| - [Forest vs Lotus feature comparison](./index.md) | ||
| - [RPC performance Grafana dashboard](https://monitoring.chain.love/public-dashboards/229eaf7ce0224655abe26c288aab914b?orgId=1&refresh=5m&kiosk) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| sidebar_position: 3 | ||
| title: Snapshot Generation Comparison | ||
| --- | ||
|
|
||
| # Snapshot Generation: Forest vs Lotus | ||
|
|
||
| Forest plays an important role in the timely, efficient generation of network snapshots. Lotus can also generate a snapshot, but doing so is significantly slower and more expensive. On a regular machine, Forest completes a basic chain snapshot export much faster than Lotus and with a fraction of the memory. | ||
|
|
||
| <p align="center"> | ||
| <img | ||
| src="/img/reports/snapshot_comparison/snapshot-export-comparison.png" | ||
| alt="Snapshot export duration: Forest vs Lotus" | ||
| style={{ maxWidth: '480px', width: '100%' }} | ||
| /> | ||
| </p> | ||
|
|
||
| Both implementations produce compatible snapshots and can consume snapshots produced by the other. | ||
|
|
||
| :::note | ||
|
sudo-shashank marked this conversation as resolved.
|
||
| Lotus snapshots are not compressed by default, whereas Forest compresses them during export. Both implementations can consume compressed snapshots. Compressing a Lotus snapshot after generation is possible, but it adds further to the total time. | ||
|
|
||
| Forest snapshots (`.forest.car.zst`) also embed an index that lets Forest use the file directly as a read-only blockstore, serving blocks in place without importing them into a database. The index is stored in skippable frames, so Lotus and other tools can still read the snapshot; they just ignore the index. Reading a snapshot in place this way is currently Forest-only. | ||
| ::: | ||
|
sudo-shashank marked this conversation as resolved.
|
||
|
|
||
| ## Related | ||
|
|
||
| - [RPC Performance Comparison](./rpc_comparison.md) | ||
| - [Forest vs Lotus feature comparison](./index.md) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't name Forest and Lotus binaries, no need to do that for Venus, no?