detect/thresh: expose threshold hash bucket depth stats#15098
Closed
detect/thresh: expose threshold hash bucket depth stats#15098
Conversation
jlucovsky
requested review from
a team,
jufajardini and
victorjulien
as code owners
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #15098 +/- ##
========================================
Coverage 82.61% 82.62%
========================================
Files 990 990
Lines 271581 271680 +99
========================================
+ Hits 224375 224476 +101
+ Misses 47206 47204 -2
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
WARNING:
Pipeline = 30522 |
| * \retval the counter id for the newly registered counter, or the already | ||
| * present counter on success | ||
| * \retval 0 on failure | ||
| * \param name Counter name (must outlive the counter). |
Member
There was a problem hiding this comment.
can this counters api stuff go into a separate commit?
Add StatsRegisterGlobalCounterWithContext() which passes a caller- supplied context pointer to the getter function on each poll. This allows multiple independent subsystems to share a single getter implementation without requiring per-instance static wrappers. Refactor counter registration to use a shared StatsFindOrAllocCounter() helper, eliminating duplication between the standard and context-aware registration paths. Move type assignment into the helper and fix a double strrchr() call when computing short_name. Issue: 8401
Add per-bucket length tracking, a nonempty_buckets atomic, and a 256-slot depth histogram to THashTableContext. The histogram enables amortized O(1) computation of the current maximum bucket depth — the value decreases as entries are removed, unlike a high-water mark. THashBucketInsert/THashBucketRemove helpers consolidate bookkeeping across all insert and remove sites. The walk-down path that lowers max_bucket_depth re-checks the vacated histogram slot before the CAS to avoid underreporting when a concurrent insert repopulates it. Issue: 8401
Register detect.thresholds.max_bucket_depth and detect.thresholds.avg_bucket_depth as global counters in ThresholdRegisterGlobalCounters(), where stats_ctx is guaranteed to be initialized. Together, avg_bucket_depth shows overall collision pressure while max_bucket_depth identifies pathological hot-spot buckets, helping determine whether the hash function or table size needs tuning. Update EVE JSON schema with the new fields. Issue: 8401
Add a note to the threshold hash-size configuration section explaining how the new avg_bucket_depth and max_bucket_depth counters can guide hash-size tuning. Issue: 8401
Contributor
Author
|
Continued in #15110 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuation of #15093
Add two new stats counters that report collision pressure in the threshold hash table:
These counters allow operators to determine whether detect.thresholds.hash-size needs tuning. A consistently high
max_bucket_depth indicates hash collisions are degrading lookup performance. A low avg_bucket_depth with a high
max indicates skewed distribution into hot-spot buckets.
Link to ticket: https://redmine.openinfosecfoundation.org/issues/8401
Describe changes:
Updates
Provide values to any of the below to override the defaults.
link to the pull request in the respective
_BRANCHvariable.SV_REPO=
SV_BRANCH=OISF/suricata-verify#2985
SU_REPO=
SU_BRANCH=