Skip to content

enhancement(core): switch from SharedTagSet to TagSet for core data model#1268

Open
tobz wants to merge 2 commits intomainfrom
tobz/core-data-model-sharedtagset-to-tagset
Open

enhancement(core): switch from SharedTagSet to TagSet for core data model#1268
tobz wants to merge 2 commits intomainfrom
tobz/core-data-model-sharedtagset-to-tagset

Conversation

@tobz
Copy link
Member

@tobz tobz commented Mar 26, 2026 •
edited
Loading

Summary

This PR switches all primary usages of SharedTagSet over to TagSet in preparation for supporting mutable tagsets on core data types.

In #1257, we added support for efficient mutable tagsets by updating TagSet to look something more like SharedTagSet: structural sharing of base tags, with additions/removals layered on top, letting us avoiding having to re-allocate entire tagsets just to add or remove a single tag. In this PR, we're switching over to actually using it.

One optimization we've had to make here is adjusting TagSet to slim down the struct size: as is, the jump in struct size between SharedTagSet and TagSet bumps up memory usage in a noticeable way even when no mutations are taking place. We switched to an on-demand design where the "overlay" data (additions/removals on top of the base tags) sits behind a heap allocation, making TagSet only 8 bytes larger than SharedTagSet. This keeps things very close, close enough to satisfy our performance/resource usage concerns.

We've also made one change in the Host Tags transform to get things compiling. In a follow up PR, we'll actually by exposing the ability to mutate tags through all the existing usages (like Context) where we'll clean up the change made in Host Tags transform.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

Existing unit tests.

References

AGTMETRICS-400

@dd-octo-sts dd-octo-sts bot added area/core Core functionality, event model, etc. area/components Sources, transforms, and destinations. destination/dogstatsd-stats DogStatsD Statistics destination. transform/host-tags Host Tags synchronous transform. encoder/datadog-traces Datadog Traces encoder. source/otlp OTLP source. transform/apm-stats APM Stats transform. labels Mar 26, 2026
@pr-commenter
Copy link

pr-commenter bot commented Mar 26, 2026 •
edited by tobz
Loading

Regression Detector Results

Run ID: 54ae9e15-7973-4bf1-bb93-38f212fec53c

Baseline: b0ed855
Comparison: 7387a9a
Diff

Optimization Goals: ❌ Regression(s) detected

perf experiment goal Δ mean % Δ mean % CI trials links
❌ otlp_ingest_metrics_5mb_cpu % cpu utilization +11.91 [+2.93, +20.88] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_ultraheavy memory utilization +8.33 [+8.19, +8.48] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_heavy memory utilization +7.26 [+7.10, +7.41] 1 (metrics) (profiles) (logs)
❌ otlp_ingest_metrics_5mb_memory memory utilization +7.15 [+6.91, +7.40] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_medium memory utilization +6.05 [+5.85, +6.24] 1 (metrics) (profiles) (logs)
❌ dsd_uds_500mb_3k_contexts_throughput ingress throughput -8.39 [-8.52, -8.26] 1 (metrics) (profiles) (logs)

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
➖ otlp_ingest_logs_5mb_memory memory utilization +4.12 [+3.61, +4.63] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_throughput ingress throughput -0.01 [-0.14, +0.12] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_cpu % cpu utilization -2.60 [-7.58, +2.38] 1 (metrics) (profiles) (logs)

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ dsd_uds_512kb_3k_contexts_cpu % cpu utilization +12.13 [-47.88, +72.14] 1 (metrics) (profiles) (logs)
❌ otlp_ingest_metrics_5mb_cpu % cpu utilization +11.91 [+2.93, +20.88] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_ultraheavy memory utilization +8.33 [+8.19, +8.48] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_heavy memory utilization +7.26 [+7.10, +7.41] 1 (metrics) (profiles) (logs)
❌ otlp_ingest_metrics_5mb_memory memory utilization +7.15 [+6.91, +7.40] 1 (metrics) (profiles) (logs)
❌ quality_gates_rss_dsd_medium memory utilization +6.05 [+5.85, +6.24] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_memory memory utilization +4.12 [+3.61, +4.63] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_memory memory utilization +3.86 [+3.68, +4.04] 1 (metrics) (profiles) (logs)
➖ dsd_uds_512kb_3k_contexts_memory memory utilization +3.75 [+3.58, +3.93] 1 (metrics) (profiles) (logs)
➖ dsd_uds_1mb_3k_contexts_memory memory utilization +3.65 [+3.48, +3.82] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_cpu % cpu utilization +3.55 [-2.84, +9.93] 1 (metrics) (profiles) (logs)
➖ dsd_uds_1mb_3k_contexts_cpu % cpu utilization +3.51 [-51.66, +58.69] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_memory memory utilization +3.35 [+3.16, +3.54] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_dsd_low memory utilization +3.34 [+3.15, +3.54] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_memory memory utilization +3.03 [+2.78, +3.28] 1 (metrics) (profiles) (logs)
➖ dsd_uds_500mb_3k_contexts_memory memory utilization +2.78 [+2.62, +2.95] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_memory memory utilization +2.78 [+2.44, +3.11] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_memory memory utilization +1.59 [+1.34, +1.85] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_idle memory utilization +0.71 [+0.69, +0.73] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_cpu % cpu utilization +0.45 [-2.08, +2.99] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_metrics_5mb_throughput ingress throughput +0.01 [-0.12, +0.14] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_throughput ingress throughput +0.01 [-0.12, +0.14] 1 (metrics) (profiles) (logs)
➖ dsd_uds_1mb_3k_contexts_throughput ingress throughput +0.00 [-0.05, +0.06] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_throughput ingress throughput +0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_throughput ingress throughput +0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_throughput ingress throughput -0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ dsd_uds_512kb_3k_contexts_throughput ingress throughput -0.00 [-0.05, +0.05] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_throughput ingress throughput -0.01 [-0.14, +0.12] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_throughput ingress throughput -0.01 [-0.04, +0.02] 1 (metrics) (profiles) (logs)
➖ dsd_uds_500mb_3k_contexts_cpu % cpu utilization -0.40 [-1.85, +1.05] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_cpu % cpu utilization -0.41 [-2.56, +1.74] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_cpu % cpu utilization -0.62 [-2.86, +1.61] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_cpu % cpu utilization -0.71 [-31.53, +30.12] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_cpu % cpu utilization -2.60 [-7.58, +2.38] 1 (metrics) (profiles) (logs)
❌ dsd_uds_500mb_3k_contexts_throughput ingress throughput -8.39 [-8.52, -8.26] 1 (metrics) (profiles) (logs)

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
✅ quality_gates_rss_dsd_heavy memory_usage 10/10 125.16MiB ≤ 140MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_low memory_usage 10/10 34.59MiB ≤ 50MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_medium memory_usage 10/10 56.10MiB ≤ 75MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_ultraheavy memory_usage 10/10 188.94MiB ≤ 200MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_idle memory_usage 10/10 21.08MiB ≤ 40MiB (metrics) (profiles) (logs)

@tobz tobz changed the title enhancement(core): switch from SharedTagSet to TagSet for core data model enhancement(core): switch from SharedTagSet to TagSet for core data model Mar 26, 2026
@pr-commenter
Copy link

pr-commenter bot commented Mar 26, 2026

Binary Size Analysis (Agent Data Plane)

Target: b0ed855 (baseline) vs 1248ff9 (comparison) diff
Analysis Type: Stripped binaries (debug symbols excluded)
Baseline Size: 26.18 MiB
Comparison Size: 26.21 MiB
Size Change: +34.54 KiB (+0.13%)
Pass/Fail Threshold: +5%
Result: PASSED ✅

Changes by Module
Module File Size Symbols
saluki_context::tags::tagset +7.20 KiB 18
saluki_components::transforms::host_tags +5.90 KiB 2
core +4.18 KiB 153
agent_data_plane::state::metrics +3.99 KiB 3
[sections] +3.66 KiB 5
saluki_context::hash::hash_context +2.87 KiB 1
axum +2.66 KiB 2
saluki_context::hash::hash_context_with_seen -1.99 KiB 1
saluki_components::destinations::dsd_stats +1.99 KiB 1
saluki_components::sources::otlp +1.74 KiB 5
saluki_env::workload::metadata -1.54 KiB 1
saluki_context::resolver::TagsResolver +1.05 KiB 8
saluki_env::workload::collectors +946 B 7
saluki_env::workload::stores -811 B 2
saluki_context::context::Context -797 B 3
saluki_components::transforms::apm_stats +793 B 4
saluki_env::workload::aggregator +765 B 1
smallvec -753 B 5
saluki_core::observability::metrics +702 B 1
saluki_components::common::otlp +665 B 6

Detailed Symbol Changes

 
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.1% +25.9Ki  +1.3% +23.0Ki    [732 Others]
  [NEW] +22.7Ki  [NEW] +22.6Ki    saluki_components::sources::otlp::logs::transform::transform_log_record::h510ca1204d3ade65
  [NEW] +10.6Ki  [NEW] +10.5Ki    <saluki_core::data_model::event::Event as core::clone::Clone>::clone.10069
  [NEW] +5.52Ki  [NEW] +5.38Ki    <hickory_proto::rr::record_data::RData as core::clone::Clone>::clone.8526
  +103% +5.34Ki  +108% +5.34Ki    _<saluki_components::transforms::host_tags::HostTagsEnrichment as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h8298bc763a5b1ed2
  [NEW] +5.13Ki  [NEW] +5.00Ki    <hickory_proto::rr::record_data::RData as core::clone::Clone>::clone.11211
  [NEW] +4.71Ki  [NEW] +4.59Ki    <rustls::error::Error as core::clone::Clone>::clone.10047
  [NEW] +4.71Ki  [NEW] +4.59Ki    <rustls::error::Error as core::clone::Clone>::clone.11226
  [NEW] +3.87Ki  [NEW] +3.76Ki    saluki_components::common::otlp::attributes::tags_from_attributes::h510bddeea173bf1d
  [NEW] +3.83Ki  [NEW] +3.71Ki    <webpki::error::Error as core::fmt::Debug>::fmt.11404
  [NEW] +3.70Ki  [NEW] +3.60Ki    saluki_context::resolver::TagsResolver::create_tag_set::h85bfa90ac5da346a
  [NEW] +3.66Ki  [NEW] +3.54Ki    <rustls::error::Error as core::fmt::Debug>::fmt.9678
  [DEL] -3.66Ki  [DEL] -3.54Ki    <rustls::error::Error as core::fmt::Debug>::fmt.9672
  [DEL] -3.83Ki  [DEL] -3.71Ki    <webpki::error::Error as core::fmt::Debug>::fmt.11397
  [DEL] -3.94Ki  [DEL] -3.83Ki    saluki_components::common::otlp::attributes::tags_from_attributes::h2b53e1f4f1fdf965
  [DEL] -4.71Ki  [DEL] -4.59Ki    <rustls::error::Error as core::clone::Clone>::clone.11219
  [DEL] -4.71Ki  [DEL] -4.59Ki    <rustls::error::Error as core::clone::Clone>::clone.10041
  [DEL] -5.13Ki  [DEL] -5.00Ki    <hickory_proto::rr::record_data::RData as core::clone::Clone>::clone.11204
  [DEL] -5.52Ki  [DEL] -5.38Ki    <hickory_proto::rr::record_data::RData as core::clone::Clone>::clone.8520
  [DEL] -10.7Ki  [DEL] -10.6Ki    <saluki_core::data_model::event::Event as core::clone::Clone>::clone.10063
  [DEL] -22.9Ki  [DEL] -22.8Ki    saluki_components::sources::otlp::logs::transform::transform_log_record::h1731e541914d182a
  +0.1% +34.5Ki  +0.1% +31.5Ki    TOTAL

@pr-commenter
Copy link

pr-commenter bot commented Mar 26, 2026

Regression Detector (Agent Data Plane)

Regression Detector Results

Run ID: e94fc2c6-e161-4522-8683-9ea7a22d8488

Baseline: b0ed855
Comparison: 1248ff9
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
➖ otlp_ingest_logs_5mb_cpu % cpu utilization +2.42 [-2.60, +7.44] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_throughput ingress throughput +0.00 [-0.13, +0.13] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_memory memory utilization -2.24 [-2.72, -1.76] 1 (metrics) (profiles) (logs)

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ dsd_uds_1mb_3k_contexts_cpu % cpu utilization +3.80 [-51.61, +59.21] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_cpu % cpu utilization +3.48 [-2.92, +9.88] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_cpu % cpu utilization +2.42 [-2.60, +7.44] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_cpu % cpu utilization +1.99 [-0.56, +4.54] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_metrics_5mb_memory memory utilization +1.91 [+1.74, +2.07] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_cpu % cpu utilization +1.16 [-0.95, +3.26] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_dsd_heavy memory utilization +0.97 [+0.84, +1.11] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_cpu % cpu utilization +0.87 [-1.40, +3.15] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_dsd_ultraheavy memory utilization +0.73 [+0.60, +0.86] 1 (metrics) (profiles) (logs)
➖ dsd_uds_1mb_3k_contexts_memory memory utilization +0.45 [+0.28, +0.62] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_memory memory utilization +0.36 [+0.18, +0.54] 1 (metrics) (profiles) (logs)
➖ dsd_uds_512kb_3k_contexts_memory memory utilization +0.36 [+0.18, +0.53] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_dsd_low memory utilization +0.35 [+0.16, +0.54] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_dsd_medium memory utilization +0.02 [-0.18, +0.21] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_throughput ingress throughput +0.01 [-0.03, +0.05] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_throughput ingress throughput +0.00 [-0.13, +0.13] 1 (metrics) (profiles) (logs)
➖ dsd_uds_512kb_3k_contexts_throughput ingress throughput +0.00 [-0.05, +0.05] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_throughput ingress throughput +0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_throughput ingress throughput +0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ dsd_uds_1mb_3k_contexts_throughput ingress throughput +0.00 [-0.06, +0.06] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_throughput ingress throughput -0.00 [-0.02, +0.02] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_metrics_5mb_throughput ingress throughput -0.01 [-0.13, +0.12] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_throughput ingress throughput -0.02 [-0.15, +0.12] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_filtering_5mb_memory memory utilization -0.02 [-0.35, +0.32] 1 (metrics) (profiles) (logs)
➖ dsd_uds_100mb_3k_contexts_memory memory utilization -0.04 [-0.22, +0.14] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_5mb_memory memory utilization -0.13 [-0.38, +0.11] 1 (metrics) (profiles) (logs)
➖ quality_gates_rss_idle memory utilization -0.18 [-0.23, -0.13] 1 (metrics) (profiles) (logs)
➖ dsd_uds_500mb_3k_contexts_memory memory utilization -0.20 [-0.37, -0.03] 1 (metrics) (profiles) (logs)
➖ dsd_uds_500mb_3k_contexts_cpu % cpu utilization -0.48 [-2.00, +1.04] 1 (metrics) (profiles) (logs)
➖ dsd_uds_512kb_3k_contexts_cpu % cpu utilization -0.53 [-57.40, +56.35] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_traces_ottl_transform_5mb_memory memory utilization -0.72 [-0.97, -0.48] 1 (metrics) (profiles) (logs)
➖ dsd_uds_10mb_3k_contexts_cpu % cpu utilization -0.93 [-32.59, +30.73] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_logs_5mb_memory memory utilization -2.24 [-2.72, -1.76] 1 (metrics) (profiles) (logs)
➖ otlp_ingest_metrics_5mb_cpu % cpu utilization -2.77 [-10.41, +4.87] 1 (metrics) (profiles) (logs)
➖ dsd_uds_500mb_3k_contexts_throughput ingress throughput -4.34 [-4.46, -4.23] 1 (metrics) (profiles) (logs)

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
✅ quality_gates_rss_dsd_heavy memory_usage 10/10 111.55MiB ≤ 140MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_low memory_usage 10/10 33.73MiB ≤ 50MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_medium memory_usage 10/10 53.05MiB ≤ 75MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_dsd_ultraheavy memory_usage 10/10 168.67MiB ≤ 200MiB (metrics) (profiles) (logs)
✅ quality_gates_rss_idle memory_usage 10/10 21.07MiB ≤ 40MiB (metrics) (profiles) (logs)

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

@tobz tobz added the type/enhancement An enhancement in functionality or support. label Mar 26, 2026
@tobz tobz marked this pull request as ready for review March 26, 2026 23:13
@tobz tobz requested a review from a team as a code owner March 26, 2026 23:13
@tobz Graphite App
Copy link
Member Author

tobz commented Mar 27, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

@tobz tobz mentioned this pull request Mar 27, 2026
Draft
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/components Sources, transforms, and destinations. area/core Core functionality, event model, etc. destination/dogstatsd-stats DogStatsD Statistics destination. encoder/datadog-traces Datadog Traces encoder. source/otlp OTLP source. transform/apm-stats APM Stats transform. transform/host-tags Host Tags synchronous transform. type/enhancement An enhancement in functionality or support.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tobz