Conversation
WalkthroughA new garbage collection tuner dependency is added to the module and imported in the core engine via a blank import to activate its initialization side-effects. No functional logic modifications are present. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
Signed-off-by: Dwi Siswanto <git@dw1.io>
dwisiswant0
force-pushed
the
dwisiswant0/feat/autotuning-gogc
branch
from
970213c to
323930c
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@go.mod`:
- Line 127: The dependency go.dw1.io/x/exp/gctuner is referenced as a
pseudo-version and via a vanity domain; update go.mod to point to a stable,
tagged module or a canonical mirror (e.g., the original repo or a maintained
fork) instead of the pseudo-version, or add a replace directive to a stable
github.com path to ensure reproducible builds and proxy availability; also audit
any runtime usage of Tuning() in your codebase and document its GOGC override
behavior in README/deployment notes so operators know the environment variable
will be changed, and consider adding a fallback or feature flag to disable GC
tuning in environments where the vanity domain or auto-tuning is undesirable.
🧹 Nitpick comments (1)
pkg/core/engine.go (1)
8-8: Consider making GC autotuning configurable for SDK users.The blank import unconditionally enables GC autotuning for all consumers of the
corepackage. SinceEngineis exported and used by both CLI (internal/runner/runner.go) and SDK users (lib/sdk_private.go,lib/multi.go,internal/server/nuclei_sdk.go), embedders may have their own GC tuning requirements or memory constraints that conflict with this automatic behavior.Consider:
- Adding a build tag (e.g.,
//go:build !nogctuner) to make the import optional- Adding a runtime configuration field to
types.Optionsto control GC tuning per Engine instance- Moving this to CLI-only initialization if it's not required for SDK use
| github.com/yassinebenaid/godump v0.11.1 | ||
| github.com/zmap/zgrab2 v0.1.8 | ||
| gitlab.com/gitlab-org/api/client-go v0.130.1 | ||
| go.dw1.io/x/exp/gctuner v0.0.0-20260204023956-e410dfed9099 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check module proxy and go.sum for gctuner dependency
echo "=== Checking module proxy for gctuner versions ==="
curl -s "https://proxy.golang.org/go.dw1.io/x/exp/gctuner/@v/list" 2>&1
echo -e "\n=== Checking go.sum for gctuner ==="
grep "gctuner" go.sum 2>/dev/null || echo "No gctuner entries found in go.sum"
echo -e "\n=== Checking go.mod for gctuner ==="
grep -A 1 -B 1 "gctuner" go.modRepository: projectdiscovery/nuclei
Length of output: 555
🌐 Web query:
gctuner go.dw1.io documentation behavior
💡 Result:
For the Go gctuner package (as documented on the Go package docs that go.dw1.io commonly mirrors), the key runtime behavior is:
- Purpose: dynamically adjusts Go’s GC aggressiveness (
GOGC/GCPercent) based on current memory usage so GC runs less often when there’s plenty of headroom, and more aggressively when memory is tight. [1] - Core formula/idea: it targets a heap “threshold” and derives a
gcPercentroughly as
gcPercent = (threshold - inuse) / inuse * 100;
ifthreshold < 2*inusethengcPercent < 100(more aggressive GC), and ifthreshold > 2*inusethengcPercent > 100(less aggressive GC). [1] Tuning(threshold uint64)behavior:- sets the heap threshold that the tuner tries to respect;
- disables tuning when
threshold == 0; - when tuning is enabled, the
GOGCenvironment variable is not honored (the tuner overrides it at runtime). [1]
- Bounds: you can set min/max GC percent via
SetMinGCPercent/SetMaxGCPercent(and query them), which the tuner uses to clamp its adjustments. [1] - Rule-of-thumb in docs: choose
thresholdaround 70% of the memory limit (host/cgroup). [1]
Sources
[1] Go package documentation for github.com/bytedance/gopkg/util/gctuner (README + function docs). (pkg.go.dev)
Citations:
🌐 Web query:
go.dw1.io x/exp gctuner garbage collector tuning
💡 Result:
go.dw1.io/x/exp/gctuner is a lightweight Go garbage-collector (GC) auto-tuner that dynamically adjusts GOGC / GCPercent so the runtime’s GC trigger stays near a chosen heap “threshold” (a target heap usage in bytes). It’s intended to keep memory usage under control while avoiding overly-frequent GC when you have plenty of headroom. [1]
How it tunes GC
Go triggers GC roughly when:
gc_trigger = heap_live + heap_live * GCPercent / 100
gctuner continuously solves for GCPercent so gc_trigger ≈ threshold, while respecting configurable min/max bounds. [1]
Install
go get -u go.dw1.io/x/exp/gctuner@latest[1]
Typical usage (recommended: 70% of effective memory limit)
import "go.dw1.io/x/exp/gctuner"
gctuner.MustEnable(
int64(gctuner.GetMemLimitPercent(70)),
gctuner.WithMinGCPercent(50),
gctuner.WithMaxGCPercent(500),
)[1]
Threshold modes
Enable(0)disables the tunerEnable(-1)derives threshold from the effective memory limitEnable(n>0)usesnbytes as the threshold [1]
Memory-limit behavior (Go 1.19+)
- Effective limit uses
GOMEMLIMITif set, otherwise host/cgroup detection. - If you call
SetMemLimitPercent(p), the tuner also sets Go’s runtime memory limit viadebug.SetMemoryLimit, with precedence:SetMemLimitPercentoverrideGOMEMLIMITthreshold[1]
Auto-enable on import
import _ "go.dw1.io/x/exp/gctuner/auto"Enables during init using package defaults. [1]
Origin
This package is a fork of github.com/bytedance/gopkg/util/gctuner. [1][2]
Sources: go.dw1.io gctuner docs [1], upstream Bytedance gctuner docs [2].
Address pseudo-version dependency and lack of official Go module proxy availability.
The go.dw1.io/x/exp/gctuner dependency is a documented garbage-collector auto-tuner (forked from ByteDance's github.com/bytedance/gopkg/util/gctuner) that dynamically adjusts GOGC based on memory thresholds. However, this module:
- Uses a pseudo-version rather than a tagged release, which may affect reproducibility if the commit becomes unavailable.
- Is not available on the official Go module proxy—it uses a personal/vanity domain (
go.dw1.io). Consider whether a mirror or fallback is needed. - When enabled via
Tuning(), overrides theGOGCenvironment variable at runtime, so ensure this behavior aligns with your deployment strategy.
Verify that the vanity domain remains stable and consider documenting the GC tuning behavior for maintainability.
🤖 Prompt for AI Agents
In `@go.mod` at line 127, The dependency go.dw1.io/x/exp/gctuner is referenced as
a pseudo-version and via a vanity domain; update go.mod to point to a stable,
tagged module or a canonical mirror (e.g., the original repo or a maintained
fork) instead of the pseudo-version, or add a replace directive to a stable
github.com path to ensure reproducible builds and proxy availability; also audit
any runtime usage of Tuning() in your codebase and document its GOGC override
behavior in README/deployment notes so operators know the environment variable
will be changed, and consider adding a fallback or feature flag to disable GC
tuning in environments where the vanity domain or auto-tuning is undesirable.
| "github.com/projectdiscovery/nuclei/v3/pkg/output" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/types" | ||
| _ "go.dw1.io/x/exp/gctuner/auto" |
There was a problem hiding this comment.
Maybe this can be implemented in https://github.com/projectdiscovery/utils and potentially applied to all tools?
There was a problem hiding this comment.
Maybe this can be implemented in https://github.com/projectdiscovery/utils and potentially applied to all tools?
Yeah, we can technically just copy it over.
but, I really need your take on this, because I'm still not 100% sure about the decision (that's why I put it in exp).
The bigger question for me is whether this behavior should also be applied to SDK consumers? Because if we decide to apply it at the SDK level (too), then this isn't just a local optimization anymore, meaning it effectively becomes a global tuning change for any program/service that depends on nuclei via SDK, also meaning that downstream users would inherit this GC behavior automatically, [which could be beneficial], but it also changes perf characteristics in a way they might not explicitly opt into.
tl;dr: whether it's appropriate to enforce this kind of dynamic GC tuning across both CLI and SDK integrations.
Really want feedback here.
2 participants
Proposed changes
Autotuning GOGC.
Rationale: put this in the
corepkg since it's the entrypoint for both CLI + SDK, so we only need to hook it up once.Proof
N/A
Checklist
Summary by CodeRabbit