feat(httptrace): add query string parameter allowlist via DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST by genesor · Pull Request #4562 · DataDog/dd-trace-go

genesor · 2026-03-18T21:26:01Z

What does this PR do?

Adds a new environment variable DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST that lets users specify a comma-separated list of query parameter keys to retain in the http.url span tag. When set, only the listed keys are kept and the expensive default obfuscation regex is bypassed entirely in favor of a simple split-and-filter pass over the raw query string.

Additionally, two side-specific env vars are introduced for granular control:

DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST_CLIENT — overrides the global allowlist for client-side (outgoing) HTTP spans only.
DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST_SERVER — overrides the global allowlist for server-side (incoming) HTTP spans only.

The global env var populates both client and server allowlists. The specific env vars take precedence when set.

Motivation

A customer on dd-trace-go v2.5 reported ~70% CPU overhead caused by the default query string obfuscation regex in URLFromRequest running on every HTTP span. Their query strings are large. Disabling query strings entirely (DD_TRACE_HTTP_URL_QUERY_STRING_DISABLED=true) was not viable since they need specific parameters for debugging. The customer requested the ability to pick specific query parameter keys to keep (e.g. [p1, p3]) with high performance.

Related: APMS-18777

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
New code is free of linting errors. You can check this by running make lint locally.
New code doesn't break existing tests. You can check this by running make test locally.
Add an appropriate team label so this PR gets put in the right place for the release notes.
All generated files are up to date. You can check this by running make generate locally.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally.

datadog-prod-us1-4 · 2026-03-18T21:31:25Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 94.87%
• Overall Coverage: 60.10%

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f0b4f60 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

codecov · 2026-03-18T21:31:53Z

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.72%. Comparing base (ee8dce2) to head (d6ad818).

Files with missing lines	Patch %	Lines
instrumentation/httptrace/config.go	90.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
instrumentation/httptrace/httptrace.go	`91.21% <100.00%> (ø)`
instrumentation/httptrace/config.go	`89.28% <90.00%> (ø)`

... and 435 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2026-03-18T21:46:30Z

Benchmarks

Benchmark execution time: 2026-04-08 16:23:30

Comparing candidate commit f0b4f60 in PR branch ben.db/feat-query-string-allowlist with baseline commit e36675b in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 215 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

hsiwikxperi · 2026-03-23T06:00:30Z

Hey @genesor
May I ask what is the progress on that issue?

genesor · 2026-03-24T10:37:20Z

Hey @genesor May I ask what is the progress on that issue?

Hi @hsiwikxperi I was OOO most of last week. I will pick it up and have it included in the next release of the tracer 👍

hsiwikxperi · 2026-03-25T19:23:42Z

Hey @genesor May I ask what is the progress on that issue?

Hi @hsiwikxperi I was OOO most of last week. I will pick it up and have it included in the next release of the tracer 👍

May I ask when you anticipate the next release?

mtoffl01

Smart!

genesor · 2026-03-25T21:53:52Z

Hey @genesor May I ask what is the progress on that issue?

Hi @hsiwikxperi I was OOO most of last week. I will pick it up and have it included in the next release of the tracer 👍

May I ask when you anticipate the next release?

We've just release the v2.7.0 on monday. The v2.8.0 should land in ~3 weeks. If there is some kind of urgency on your side I could see if a patch release is possible on v2.7.1.

Another question, this modification applies to both client and server queries. Is that okay for you ? If not I'd need to introduce two new settings (one for client and one for server, taking precedence over the global one)

WDYT ?

hsiwikxperi · 2026-03-26T06:57:04Z

Hey @genesor,
thank you for you question.
Do you mean that the env variable will be applied to all wrappers the same way? Would it be possible to have sth like in this pseudo-Go code :)

client1 := httptracer.WrapClient(WithQueryAllowList(allowList1))
client2 := httptracer.WrapClient(WithQueryAllowList(allowList2))

srvMux := httptrace.NewServeMux(WithQueryAllowList(allowList3))

genesor · 2026-03-26T09:59:46Z

Hi @hsiwikxperi

Having "per client" allowlist require some more plumbing, especially with the programmatic option. The allowlist is consumed deep in URLFromRequest() which reads from global http config and we have nothing in place yet to have per-wrapper configuration.

I will explore options to find a nice way to make it happen.

In the meantime would it be okay for you to have three env var:

DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST global default for both client & server
DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST_CLIENT client-specific override (takes precedence over global)
DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST_SERVER server-specific override (takes precedence over global)

hsiwikxperi · 2026-03-27T06:21:35Z

Sure, for now it should be sufficient, however, I would keep in mind configuration per HTTP wrapper.

…TP_URL_QUERY_STRING_ALLOWLIST

…ents

…orted configurations

…v vars

… allowlist configs

darccio · 2026-03-27T17:16:56Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6ad818387

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

darccio · 2026-03-27T17:41:00Z

+		})
+	}
+}
+


Provided that CPU overhead is the origin of this PR, could we have a benchmark comparing the new implementation against the previous?

I've added a benchmark in b1ee83a

BenchmarkURLFromRequest BenchmarkURLFromRequest/regex/few_params BenchmarkURLFromRequest/regex/few_params-16 156728 7656 ns/op 186 B/op 6 allocs/op BenchmarkURLFromRequest/allowlist/few_params BenchmarkURLFromRequest/allowlist/few_params-16 10313593 117.2 ns/op 72 B/op 3 allocs/op BenchmarkURLFromRequest/regex/many_params BenchmarkURLFromRequest/regex/many_params-16 50014 24293 ns/op 447 B/op 7 allocs/op BenchmarkURLFromRequest/allowlist/many_params BenchmarkURLFromRequest/allowlist/many_params-16 5144888 238.5 ns/op 120 B/op 4 allocs/op BenchmarkURLFromRequest/regex/really_long_1 BenchmarkURLFromRequest/regex/really_long_1-16 3770 316669 ns/op 4675 B/op 5 allocs/op BenchmarkURLFromRequest/allowlist/really_long_1 BenchmarkURLFromRequest/allowlist/really_long_1-16 3190782 372.1 ns/op 216 B/op 5 allocs/op BenchmarkURLFromRequest/regex/really_long_2 BenchmarkURLFromRequest/regex/really_long_2-16 3834 311909 ns/op 4418 B/op 5 allocs/op BenchmarkURLFromRequest/allowlist/really_long_2 BenchmarkURLFromRequest/allowlist/really_long_2-16 3284832 373.2 ns/op 216 B/op 5 allocs/op PASS ok github.com/DataDog/dd-trace-go/v2/instrumentation/httptrace 10.320s

@rarguelloF I ran BenchmarkURLFromRequest on main as asked. (regex
obfuscation only, current behavior)

main :

BenchmarkURLFromRequest BenchmarkURLFromRequest/regex/few_params BenchmarkURLFromRequest/regex/few_params-16 122654 9799 ns/op 185 B/op 6 allocs/op BenchmarkURLFromRequest/regex/many_params BenchmarkURLFromRequest/regex/many_params-16 17601 88960 ns/op 447 B/op 7 allocs/op BenchmarkURLFromRequest/regex/really_long_1 BenchmarkURLFromRequest/regex/really_long_1-16 2037 753900 ns/op 4576 B/op 5 allocs/op BenchmarkURLFromRequest/regex/really_long_2 BenchmarkURLFromRequest/regex/really_long_2-16 3505 331900 ns/op 4316 B/op 5 allocs/op PASS ok github.com/DataDog/dd-trace-go/v2/instrumentation/httptrace 63.928s

new:

BenchmarkURLFromRequest BenchmarkURLFromRequest/regex/few_params BenchmarkURLFromRequest/regex/few_params-16 156728 7656 ns/op 186 B/op 6 allocs/op BenchmarkURLFromRequest/regex/many_params BenchmarkURLFromRequest/regex/many_params-16 50014 24293 ns/op 447 B/op 7 allocs/op BenchmarkURLFromRequest/regex/really_long_1 BenchmarkURLFromRequest/regex/really_long_1-16 3770 316669 ns/op 4675 B/op 5 allocs/op

darccio · 2026-03-27T17:45:25Z

+	for rawQuery != "" {
+		// Split on "&" to isolate each key=value pair.
+		var pair string
+		if before, after, found := strings.Cut(rawQuery, "&"); found {


Keys and values can be URL-encoded. Maybe we should call url.QueryUnescape on rawQuery?

Not sold to the idea for two reasons:

calling url.QueryUnescape on the entire URL would break the & and then break the splitting.

calling url.QueryUnescape on each key would be a performance trap

Keys in RawQuery can indeed be percent-encoded. I've added handling for non-ASCII keys (e.g. CJK characters like 名前): parseAllowlist now stores both the raw UTF-8 form and the percent-encoded form (%E5%90%8D%E5%89%8D) in the map at config time. This way the lookup in filterQueryStringByAllowlist matches directly against the encoded key in RawQuery with zero runtime decoding cost.

For the case of percent-encoded ASCII keys (e.g. %70age for page), we intentionally skip it, no standard HTTP client encodes plain ASCII letters in parameter names

hsiwikxperi · 2026-04-08T09:22:50Z

@genesor May I ask when we can expect the release with this future?

kakkoyun · 2026-04-08T10:38:17Z

 		c.resourceRenamingEnabled = &vv
 	}
+	// Global allowlist applies to both client and server; specific env vars override it.
+	if v, ok := env.Lookup(envQueryStringAllowlist); ok && v != "" {


Just to make sure, if both DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST and DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST_CLIENT|SERVER specified together DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST will be overridden is this what we want?

We should document the presedence in any case. And add a test case for this.

Yes @kakkoyun there is a global allowlist that can be overriden by specific ones, this is what the comment states. It is also included in the registry description and the docstring in the code

Two test cases already cover this exact scenario

Okay, thanks for the pointers! I missed those. I have reviewed too many PRs today.

hsiwikxperi · 2026-04-17T08:44:43Z

@genesor
Do you happen to know when we can expect this PR to be released?

genesor · 2026-04-20T10:05:24Z

Hi @hsiwikxperi

The v2.8.0 is currently in the process of being release (https://github.com/DataDog/dd-trace-go/tree/release-v2.8.x) the v2.8.0-rc.1 is already available.

genesor marked this pull request as ready for review March 25, 2026 18:10

genesor requested review from a team as code owners March 25, 2026 18:10

genesor force-pushed the ben.db/feat-query-string-allowlist branch 2 times, most recently from 3cce1cc to 112a1e9 Compare March 25, 2026 19:47

mtoffl01 approved these changes Mar 25, 2026

View reviewed changes

genesor requested review from a team as code owners March 26, 2026 15:44

github-actions Bot added the apm:ecosystem contrib/* related feature requests or bugs label Mar 26, 2026

genesor force-pushed the ben.db/feat-query-string-allowlist branch from aa29f92 to e7b72a8 Compare March 26, 2026 15:59

genesor requested a review from mtoffl01 March 26, 2026 16:00

genesor added 4 commits March 27, 2026 14:46

feat(httptrace): add query string parameter allowlist via DD_TRACE_HT…

ca3e631

…TP_URL_QUERY_STRING_ALLOWLIST

fix(httptrace): use strings.Cut per golangci-lint and add inline comm…

fb0bc08

…ents

chore(env): register DD_TRACE_HTTP_URL_QUERY_STRING_ALLOWLIST in supp…

a8825b9

…orted configurations

feat(httptrace): add client/server-specific query string allowlist en…

54782a1

…v vars

genesor force-pushed the ben.db/feat-query-string-allowlist branch from e7b72a8 to 54782a1 Compare March 27, 2026 13:46

fix(env): set correct type and default for client/server query string…

d6ad818

… allowlist configs

chatgpt-codex-connector Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread instrumentation/httptrace/config.go

darccio reviewed Mar 27, 2026

View reviewed changes

rarguelloF approved these changes Apr 8, 2026

View reviewed changes

kakkoyun reviewed Apr 8, 2026

View reviewed changes

kakkoyun approved these changes Apr 8, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: retrying mergequeue-status: rejected and removed mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: retrying labels Apr 8, 2026

Merge branch 'main' into ben.db/feat-query-string-allowlist

f0b4f60

gh-worker-dd-mergequeue-cf854d Bot merged commit 490ac5c into main Apr 8, 2026
186 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Apr 8, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the ben.db/feat-query-string-allowlist branch April 8, 2026 16:48

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Apr 8, 2026

Conversation

genesor commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Reviewer's Checklist

Uh oh!

datadog-prod-us1-4 Bot commented Mar 18, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter Bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Uh oh!

hsiwikxperi commented Mar 23, 2026

Uh oh!

genesor commented Mar 24, 2026

Uh oh!

hsiwikxperi commented Mar 25, 2026

Uh oh!

mtoffl01 left a comment

Choose a reason for hiding this comment

Uh oh!

genesor commented Mar 25, 2026

Uh oh!

hsiwikxperi commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

genesor commented Mar 26, 2026

Uh oh!

hsiwikxperi commented Mar 27, 2026

Uh oh!

darccio commented Mar 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiwikxperi commented Apr 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hsiwikxperi commented Apr 17, 2026

Uh oh!

genesor commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

genesor commented Mar 18, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Mar 18, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

codecov Bot commented Mar 18, 2026 •

edited

Loading

pr-commenter Bot commented Mar 18, 2026 •

edited

Loading

hsiwikxperi commented Mar 26, 2026 •

edited

Loading